Text processing method and device and electronic equipment
By combining the context of sensitive words in text processing to determine candidate replacement words and their sensitivity protection index, and selecting target replacement words for replacement, the problem of missing text content and misunderstanding caused by sensitive word replacement in the prior art is solved, thus improving the efficiency of text processing.
Patent Information
- Application Number
- CN202511535435.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-27
- Publication Date
- 2026-02-24
Smart Images
Figure CN121562601A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of text processing technology, and in particular to a text processing method, apparatus and electronic device. Background Technology
[0002] Currently, the process of replacing sensitive words in text can be as follows: obtain the sensitive words in the text; mask some characters in the sensitive words to obtain the processed text.
[0003] In the above scheme, some characters in sensitive words are replaced with asterisks or other placeholders to achieve masking, resulting in missing text content, making the text content difficult to understand or causing misunderstandings, and reducing text processing efficiency. Summary of the Invention
[0004] This disclosure provides a text processing method, apparatus, and electronic device.
[0005] According to a first aspect of the present disclosure, a text processing method is provided, the method comprising: determining a text to be processed and a list of sensitive words corresponding to the text; the list of sensitive words including at least one sensitive word; for each sensitive word in the list of sensitive words, determining at least one candidate replacement word and a sensitivity protection index of the at least one candidate replacement word based on the sensitive word and the context words corresponding to the sensitive word; selecting a target replacement word corresponding to the sensitive word from the at least one candidate replacement word based on the sensitivity protection index; and performing sensitive word replacement processing on the text based on the target replacement word corresponding to the at least one sensitive word in the text to obtain processed text.
[0006] In one embodiment of this disclosure, for each sensitive word in the sensitive word list, at least one candidate replacement word is determined based on the sensitive word and the context words corresponding to the sensitive word, including: for each sensitive word in the sensitive word list, determining a reference word library that matches the part-of-speech type and / or sentiment type of the sensitive word; and determining the at least one candidate replacement word based on the sensitive word, the context words corresponding to the sensitive word, and at least one reference word in the reference word library.
[0007] In one embodiment of this disclosure, determining a reference word library matching the part-of-speech type and / or sentiment type of each sensitive word in the sensitive word list includes: determining the part-of-speech type and / or sentiment type of the sensitive word; obtaining a preset word library; selecting at least one word having the part-of-speech type and / or sentiment type from the preset word library according to the part-of-speech type and / or sentiment type of the sensitive word; and performing combination processing on the at least one word to obtain the reference word library.
[0008] In one embodiment of this disclosure, determining the at least one candidate replacement word based on the sensitive word, the context words corresponding to the sensitive word, and at least one reference word in the reference word library includes: selecting at least one preliminary candidate replacement word from the at least one reference word based on the similarity between the sensitive word and the at least one reference word; determining the context adaptation score of the at least one preliminary candidate replacement word based on the context words corresponding to the sensitive word and a context adaptation scoring strategy; and selecting the at least one candidate replacement word from the at least one preliminary candidate replacement word based on the context adaptation score.
[0009] In one embodiment of this disclosure, determining the context adaptation score of the at least one preliminary candidate replacement word based on the context words corresponding to the sensitive word and the context adaptation scoring strategy includes: for each preliminary candidate replacement word among the at least one preliminary candidate replacement words, determining the vector similarity between the preliminary candidate replacement word and each word in the context words; and determining the mean of each vector similarity as the context adaptation score of the preliminary candidate replacement word.
[0010] In one embodiment of this disclosure, for each sensitive word in the sensitive word list, a sensitivity protection index for the at least one candidate replacement word is determined based on the sensitive word and its corresponding context words. This includes: for each candidate replacement word in the at least one candidate replacement word, determining the parameter values of the candidate replacement word on at least two of the following parameters: the difference between 1 and the sensitivity probability of the candidate replacement word, the semantic similarity between the candidate replacement word and the sensitive word, and the context adaptability score of the candidate replacement word; and determining the sensitivity protection index of the candidate replacement word based on the parameter values on the at least two parameters.
[0011] In one embodiment of this disclosure, determining the sensitivity protection index of the candidate replacement word based on the parameter values of the at least two parameters includes: determining the application scenario of the text and the weights of the at least two parameters in the application scenario; and determining the sensitivity protection index of the candidate replacement word based on the at least two weights and the parameter values of the at least two parameters.
[0012] In one embodiment of this disclosure, the method for determining the sensitivity probability of the candidate replacement word includes: determining a first representation vector of the candidate replacement word; inputting the first representation vector into a word sense sensitivity model to obtain the sensitivity probability of the candidate replacement word.
[0013] In one embodiment of this disclosure, determining the text to be processed and the list of sensitive words corresponding to the text includes: determining the text to be processed; performing word segmentation on the text to obtain a word segmentation list; performing vectorization on the word segmentation words in the word segmentation list to obtain a word segmentation vector list; inputting the word segmentation vector list into a semantic sensitivity model to obtain the sensitivity probability of each word segmentation word in the word segmentation list; and selecting sensitive words from each word segmentation word in the word segmentation list according to the sensitivity probability to obtain the sensitive word list.
[0014] According to a second aspect of the present disclosure, a text processing apparatus is also provided, the apparatus comprising: a first determining module, configured to determine a text to be processed and a list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word; a second determining module, configured to, for each sensitive word in the list of sensitive words, determine at least one candidate replacement word and a sensitivity protection index of the at least one candidate replacement word based on the sensitive word and the context words corresponding to the sensitive word; a selecting module, configured to select a target replacement word corresponding to the sensitive word from the at least one candidate replacement word based on the sensitivity protection index; and a replacement processing module, configured to perform sensitive word replacement processing on the text based on the target replacement word corresponding to the at least one sensitive word in the text, to obtain processed text.
[0015] According to a third aspect of the present disclosure, an electronic device is also provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: implement the steps of the text processing method as described above.
[0016] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is also provided, which, when instructions in the storage medium are executed by a processor, enables the processor to perform the text processing method as described above.
[0017] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: The text to be processed is determined along with a list of sensitive words. This list includes at least one sensitive word. For each sensitive word in the list, at least one candidate replacement word and its sensitivity protection index are determined based on the sensitive word and its corresponding context words. Based on the sensitivity protection index, a target replacement word is selected from the at least one candidate replacement word. The text is then processed using the target replacement word corresponding to at least one sensitive word, resulting in the processed text. By combining the target replacement word with the context words corresponding to the sensitive word, the text processing avoids missing content, ensures easy comprehension, prevents misunderstandings, and improves processing efficiency.
[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.
[0020] Figure 1 This is a flowchart of a text processing method according to an embodiment of the present disclosure; Figure 2 This is a flowchart of a text processing method according to another embodiment of the present disclosure; Figure 3 This is a flowchart of a text processing method according to another embodiment of the present disclosure; Figure 4 This is a schematic diagram of the structure of a text processing apparatus according to an embodiment of the present disclosure; Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation
[0021] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.
[0022] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.
[0023] Currently, the process of replacing sensitive words in text can be as follows: obtain the sensitive words in the text; mask some characters in the sensitive words to obtain the processed text.
[0024] In the above scheme, some characters in sensitive words are replaced with asterisks or other placeholders to achieve masking, resulting in missing text content, making the text content difficult to understand or causing misunderstandings, and reducing text processing efficiency.
[0025] Figure 1 This is a flowchart of a text processing method according to an embodiment of the present disclosure. It should be noted that the text processing method of this embodiment can be applied to a text processing device, which can be configured in an electronic device to enable the electronic device to perform text processing functions.
[0026] The electronic device can be any device with computing capabilities, such as a terminal device or a server. The following embodiments use an electronic device as an example for illustration.
[0027] like Figure 1 As shown, the method includes the following steps: Step 101: Determine the text to be processed and the list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word.
[0028] In this embodiment of the disclosure, the electronic device may perform step 101 as follows: determine the text to be processed; perform word segmentation on the text to obtain a word segmentation list; perform vectorization on the word segmentation words in the word segmentation list to obtain a word segmentation vector list; input the word segmentation vector list into a word sense sensitivity model to obtain the sensitivity probability of each word segmentation word in the word segmentation list; and select sensitive words from each word segmentation word in the word segmentation list according to the sensitive word probability to obtain a sensitive word list.
[0029] Among them, the text to be processed can be text obtained from any source, or text obtained by extracting text from multimodal data obtained from any source. Among them. The source, for example, can be a database, a file, a web page, etc., and specific limitations are not made here.
[0030] In the embodiments of the present disclosure, after determining the text to be processed, in order to improve the processing efficiency and accuracy of the text, the text can be preprocessed. Among them, the preprocessing can include at least one of the following: uniformly processing the character format in the text, and removing meaningless symbols in the text.
[0031] Among them, uniformly processing the character format in the text, for example, converting full-width characters to half-width characters, converting uppercase letters to lowercase letters, etc. Among them, the meaningless symbols in the text, for example, meaningless special symbols or punctuation marks, etc.
[0032] In the embodiments of the present disclosure, an electronic device can obtain a word segmentation model; perform word segmentation processing on the text in combination with the word segmentation model to obtain a word segmentation list. Among them, the word segmentation model, for example, can be the Jieba word segmentation model, etc., and specific limitations are not made here. Among them, when the word segmentation model performs word segmentation processing on the text, it can use the dictionary自带 in the word segmentation model, and / or, it can use the dictionary provided by the electronic device to the word segmentation model.
[0033] In the embodiments of the present disclosure, after determining the word segmentation list, in order to further improve the accuracy of the word segmentation list, a stop word list can be obtained; perform stop word filtering processing on the word segmentation list according to the stop word list to filter out the stop words in the word segmentation list. Among them, the stop words, for example, are "de", "shi", "zai", etc.
[0034] In the embodiments of the present disclosure, after determining the word segmentation list, in order to further improve the accuracy of the word segmentation list, the word segmentation list can be normalized. Among them, the normalization processing, for example, uniformly processes the traditional Chinese characters and simplified Chinese characters in the word segmentation list, and uniformly processes the synonymous words in the word segmentation list, etc.
[0035] In the embodiments of the present disclosure, the process of the electronic device performing vectorization processing on the segmented words in the word segmentation list to obtain a word segmentation vector list can be, for example, obtaining a vectorization model; for each segmented word in the word segmentation list, input the segmented word into the vectorization model, and obtain the segmented vector output by the vectorization model. Among them, the vectorization model, for example, is the tokenizer network in the BERT model, the word vector model, etc.
[0036] In this embodiment of the disclosure, the semantic sensitivity model can combine one of the segmented word vectors in the segmented word vector list with the context segmented word vectors of that segmented word vector to determine the sensitivity probability of the segmented word corresponding to that segmented word vector. This allows the sensitivity probability of the segmented word to be determined by combining the context words of that segmented word.
[0037] Among them, the semantic sensitivity model can be trained by combining a large number of sensitive words and non-sensitive words.
[0038] In this embodiment of the disclosure, the electronic device can set a sensitivity probability threshold. Correspondingly, the process by which the electronic device selects sensitive words from each segmented word in the segmentation list based on the sensitivity probability to obtain a sensitive word list can be, for example, that for each segmented word in the segmentation list, if the sensitivity probability of the segmented word is greater than or equal to the sensitivity probability threshold, the segmented word is determined to be a sensitive word; if the sensitivity probability of the segmented word is less than the sensitivity probability threshold, the segmented word is determined not to be a sensitive word.
[0039] The sensitivity probability threshold can be determined based on the sensitivity requirements for the text. The sensitivity requirements are negatively correlated with the sensitivity probability threshold. For example, if the sensitivity requirements are low, a higher sensitivity probability threshold is set; if the sensitivity requirements are high, a lower sensitivity probability threshold is set.
[0040] Step 102: For each sensitive word in the list of sensitive words, determine at least one candidate replacement word and its sensitivity protection index based on the sensitive word and its corresponding context words.
[0041] In this embodiment of the disclosure, the electronic device performing step 102 may, for example, involve determining at least one candidate replacement word for each sensitive word in the sensitive word list, based on the sensitive word and its corresponding context words; and determining the sensitivity protection index of the at least one candidate replacement word based on the sensitive word and its corresponding context words. The at least one candidate replacement word may be selected from a preset word library.
[0042] The context words corresponding to the sensitive words can be words in the text whose distance from the sensitive words is less than a preset distance threshold. The word distance is the number of words separating two words.
[0043] Step 103: Based on the sensitivity protection index, select the target replacement word corresponding to the sensitive word from at least one candidate replacement word.
[0044] In this embodiment of the disclosure, the electronic device may perform step 103 as follows: sort at least one candidate replacement word in descending order according to the sensitivity protection index to obtain a sorting result; and determine the candidate replacement word ranked first in the sorting result as the target replacement word corresponding to the sensitive word.
[0045] Step 104: Based on the target replacement word corresponding to at least one sensitive word in the text, perform sensitive word replacement processing on the text to obtain the processed text.
[0046] In this embodiment of the disclosure, the electronic device may perform step 104 as follows: for each sensitive word in the text, the sensitive word is replaced by the target replacement word corresponding to the sensitive word; after all the sensitive words in the text have been replaced, the text at this time is determined as the processed text.
[0047] In the text processing method of this embodiment, the following steps are taken: First, determine the text to be processed and a list of sensitive words corresponding to the text. The list of sensitive words includes at least one sensitive word. For each sensitive word in the list, determine at least one candidate replacement word and a sensitivity protection index of the candidate replacement word based on the sensitive word and its corresponding context words. Then, select a target replacement word corresponding to the sensitive word from the at least one candidate replacement word based on the sensitivity protection index. Finally, perform sensitive word replacement processing on the text based on the target replacement word corresponding to at least one sensitive word, obtaining the processed text. By combining the target replacement word with the context words corresponding to the sensitive word and then performing the replacement processing, text content loss can be avoided during sensitive word replacement processing, ensuring that the text content is easy to understand, avoiding misunderstandings, and improving text processing efficiency.
[0048] Figure 2 This is a flowchart illustrating another embodiment of a text processing method disclosed herein. It should be noted that the text processing method of this embodiment can be applied to a text processing device, which can be configured in an electronic device to enable the electronic device to perform text processing functions.
[0049] The electronic device can be any device with computing capabilities, such as a terminal device or a server. The following embodiments use an electronic device as an example for illustration.
[0050] like Figure 2 As shown, the method includes the following steps: Step 201: Determine the text to be processed and the list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word.
[0051] Step 202: For each sensitive word in the list of sensitive words, determine a reference word library that matches the part-of-speech type and / or sentiment type of the sensitive word.
[0052] In this embodiment of the disclosure, the process of the electronic device performing step 202 may include, for example, determining the part-of-speech type and / or sentiment type of the sensitive words; obtaining a preset word library; selecting at least one word with a part-of-speech type and / or sentiment type from the preset word library according to the part-of-speech type and / or sentiment type of the sensitive words; and combining the at least one word to obtain a reference word library.
[0053] The part-of-speech type of sensitive words is classified according to their grammatical function and morphological characteristics in a sentence. The part-of-speech type of sensitive words includes, for example, nouns, verbs, adjectives, adverbs, pronouns, prepositions, conjunctions, and articles, but no specific restrictions are made here.
[0054] The sentiment type of sensitive words is a classification based on the emotional tendency expressed by the sensitive words. The sentiment type of sensitive words, such as positive sentiment, negative sentiment, neutral sentiment, and mixed sentiment, is not specifically defined here.
[0055] In one embodiment of this disclosure, the electronic device can select at least one word with a certain part-of-speech type from a preset word library based on the part-of-speech type of the sensitive word, and then combine them to obtain a reference word library. In another embodiment, the electronic device can select at least one word with a certain sentiment type from a preset word library based on both the part-of-speech type and the sentiment type of the sensitive word, and then combine them to obtain a reference word library.
[0056] Step 203: Determine at least one candidate replacement word based on the sensitive word, the context words corresponding to the sensitive word, and at least one reference word in the reference word library.
[0057] In this embodiment of the disclosure, the electronic device may perform step 203 as follows: select at least one preliminary candidate replacement word from at least one reference word based on the similarity between the sensitive word and at least one reference word; determine the context adaptation score of at least one preliminary candidate replacement word based on the context words corresponding to the sensitive word and the context adaptation scoring strategy; and select at least one candidate replacement word from at least one preliminary candidate replacement word based on the context adaptation score.
[0058] The process by which an electronic device selects at least one preliminary candidate replacement word from at least one reference word based on the similarity between a sensitive word and at least one reference word can be, for example, as follows: determining a second representation vector of the sensitive word and a third representation vector of each reference word; for each reference word, determining the vector similarity between the reference word and the sensitive word based on the third representation vector and the second representation vector; if the vector similarity is greater than or equal to a vector similarity threshold, determining the reference word as a preliminary candidate replacement word; if the vector similarity is less than the vector similarity threshold, determining the reference word as not a preliminary candidate replacement word.
[0059] The process by which the electronic device determines the context adaptation score of at least one preliminary candidate replacement word based on the context words corresponding to the sensitive word and the context adaptation scoring strategy can be, for example, by determining the vector similarity between each preliminary candidate replacement word and each word in the context words for each preliminary candidate replacement word; and determining the mean of each vector similarity as the context adaptation score of the preliminary candidate replacement word.
[0060] The process by which the electronic device determines the vector similarity between the preliminary candidate replacement word and each word in the context can be, for example, as follows: determining the fourth representation vector of the preliminary candidate replacement word; determining the fifth representation vector of each word in the context; and for each word in the context, determining the vector similarity between the word and the preliminary candidate replacement word based on the fifth and fourth representation vectors of the word.
[0061] The formula for calculating the contextual adaptability score of the preliminary candidate replacement words can be, for example, as shown in the following formula (1).
[0062] (1) in, Indicates preliminary candidate replacement words; express Contextual adaptability score; express The representation vector; This indicates the first word in the context of the sensitive word. One word; express The representation vector; Indicates the number of words in the context.
[0063] Step 204: Determine the sensitivity protection index of at least one candidate replacement word based on the sensitive words and the context words corresponding to the sensitive words.
[0064] In this embodiment of the disclosure, the electronic device can input sensitive words, the context words corresponding to the sensitive words, and candidate replacement words into the sensitivity protection index model to obtain the sensitivity protection index of the candidate replacement words.
[0065] Step 205: Based on the sensitivity protection index, select the target replacement word corresponding to the sensitive word from at least one candidate replacement word.
[0066] Step 206: Based on the target replacement word corresponding to at least one sensitive word in the text, perform sensitive word replacement processing on the text to obtain the processed text.
[0067] It should be noted that for details regarding steps 201, 204 to 206, please refer to [the relevant documentation / reference]. Figure 1 Steps 101 to 104 in the illustrated embodiment will not be described in detail here.
[0068] In the text processing method of this embodiment, the following steps are taken: First, determine the text to be processed and a list of sensitive words corresponding to the text. The list of sensitive words includes at least one sensitive word. For each sensitive word in the list, determine a reference word library that matches the part-of-speech type and / or sentiment type of the sensitive word. Second, determine at least one candidate replacement word based on the sensitive word, the context words corresponding to the sensitive word, and at least one reference word in the reference word library. Third, determine a sensitivity protection index for the at least one candidate replacement word based on the sensitive word and the context words corresponding to the sensitive word. Fourth, select a target replacement word corresponding to the sensitive word from the at least one candidate replacement word based on the sensitivity protection index. Fifth, perform sensitive word replacement processing on the text based on the target replacement word corresponding to the at least one sensitive word in the text to obtain the processed text. The step of selecting at least one candidate replacement word from the reference word library based on the sensitive word and the context words corresponding to the sensitive word ensures semantic consistency between the candidate replacement word and the text content, thereby ensuring the fluency of the processed text and further improving text processing efficiency.
[0069] Figure 3 This is a flowchart illustrating another embodiment of a text processing method disclosed herein. It should be noted that the text processing method of this embodiment can be applied to a text processing device, which can be configured in an electronic device to enable the electronic device to perform text processing functions.
[0070] The electronic device can be any device with computing capabilities, such as a terminal device or a server. The following embodiments use an electronic device as an example for illustration.
[0071] like Figure 3 As shown, the method includes the following steps: Step 301: Determine the text to be processed and the list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word.
[0072] Step 302: For each sensitive word in the list of sensitive words, determine at least one candidate replacement word based on the sensitive word and the context words corresponding to the sensitive word.
[0073] Step 303: For each candidate replacement word in at least one candidate replacement word, determine the parameter values of the candidate replacement word on at least two of the following parameters: the difference between 1 and the sensitivity probability of the candidate replacement word, the semantic similarity between the candidate replacement word and the sensitive word, and the contextual adaptability score of the candidate replacement word.
[0074] In this embodiment of the disclosure, the method for determining the sensitivity probability of the candidate replacement word may include: determining a first representation vector of the candidate replacement word; inputting the first representation vector into a word sense sensitivity model to obtain the sensitivity probability of the candidate replacement word.
[0075] The higher the difference between 1 and the sensitivity probability of the candidate replacement word, the less sensitive the candidate replacement word is.
[0076] The method for determining the semantic similarity between candidate replacement words and sensitive words may include: inputting candidate replacement words and sensitive words into a semantic similarity model and obtaining the semantic similarity output by the semantic similarity model.
[0077] The method for determining the context adaptability score of the candidate replacement word may include: determining the vector similarity between each word in the context words corresponding to the candidate replacement word and the sensitive word; and determining the mean of each vector similarity as the context adaptability score of the candidate replacement word.
[0078] Step 304: Determine the sensitivity protection index of the candidate replacement words based on the parameter values of at least two parameters.
[0079] In one embodiment of this disclosure, the electronic device can directly determine the sensitivity protection index of the candidate replacement word based on the parameter values of at least two parameters. In another embodiment, the electronic device can determine fixed weights for at least two parameters; and determine the sensitivity protection index of the candidate replacement word based on the fixed weights and the parameter values of the at least two parameters.
[0080] In another example, to ensure the accuracy of the sensitivity protection index under different application scenarios, different weights are set according to different application scenarios. Correspondingly, the electronic device performing step 304 can be, for example, determining the application scenario of the text and the weights of at least two parameters under the application scenario; and determining the sensitivity protection index of the candidate replacement words based on the at least two weights and the parameter values of the at least two parameters.
[0081] For example, in some application scenarios where semantic similarity is emphasized, the weight of this parameter can be increased. In other application scenarios where sensitivity protection is emphasized, the weight of this parameter can be increased.
[0082] The formula for calculating the sensitivity protection index of candidate replacement words can be shown in the following formula (2).
[0083] (2) in, Indicates candidate replacement words; express Sensitivity protection index; express The probability of sensitivity; Indicates sensitive words; Indicates the semantic similarity between candidate replacement words and sensitive words; express The representation vector of the i-th word in the corresponding context words; express The number of words in the corresponding context; The context-appropriate score of the candidate replacement words; , as well as Indicates the weight.
[0084] Step 305: Based on the sensitivity protection index, select the target replacement word corresponding to the sensitive word from at least one candidate replacement word.
[0085] Step 306: Based on the target replacement word corresponding to at least one sensitive word in the text, perform sensitive word replacement processing on the text to obtain the processed text.
[0086] It should be noted that for details regarding steps 301 to 302 and steps 305 to 306, please refer to [the relevant documentation / reference]. Figure 1 Steps 101 to 104 in the illustrated embodiment will not be described in detail here.
[0087] In the text processing method of this disclosure embodiment, the following steps are taken: First, determine the text to be processed and a list of sensitive words corresponding to the text. The list of sensitive words includes at least one sensitive word. For each sensitive word in the list, determine at least one candidate replacement word based on the sensitive word and its corresponding context words. For each candidate replacement word, determine the parameter values of the candidate replacement word on at least two of the following parameters: the difference between the sensitivity probability of the candidate replacement word and the sensitive word, the semantic similarity between the candidate replacement word and the sensitive word, and the context adaptability score of the candidate replacement word. Based on the parameter values on at least two parameters, determine the sensitivity guarantee of the candidate replacement word. Sensitivity protection index; based on the sensitivity protection index, select the target replacement word corresponding to the sensitive word from at least one candidate replacement word; based on the target replacement word corresponding to at least one sensitive word in the text, perform sensitive word replacement processing on the text to obtain the processed text; wherein, the sensitivity probability of the candidate replacement word, the context adaptability score, and the semantic similarity between the candidate replacement word and the sensitive word are comprehensively considered to determine the sensitivity protection index of the candidate replacement word, and then determine the target replacement word, thereby ensuring that the sensitivity probability of the target replacement word is low and that the semantic consistency between the target replacement word and the text content is ensured, thereby ensuring the fluency of the processed text content, and further improving the efficiency of text processing.
[0088] Figure 4 This is a schematic diagram of the structure of a text processing apparatus according to an embodiment of the present disclosure.
[0089] like Figure 4 As shown, the text processing device may include: a first determining module 401, a second determining module 402, a selection module 403, and a replacement processing module 404.
[0090] The system includes a first determining module 401, which determines the text to be processed and a list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word; a second determining module 402, which determines at least one candidate replacement word and a sensitivity protection index of the at least one candidate replacement word for each sensitive word in the list of sensitive words, based on the sensitive word and the context words corresponding to the sensitive word; a selecting module 403, which selects a target replacement word corresponding to the sensitive word from the at least one candidate replacement word based on the sensitivity protection index; and a replacement processing module 404, which performs sensitive word replacement processing on the text based on the target replacement word corresponding to the at least one sensitive word in the text, to obtain the processed text.
[0091] In one embodiment of this disclosure, the second determining module 402 includes: a first determining unit and a second determining unit; the first determining unit is configured to determine a reference word library that matches the part-of-speech type and / or sentiment type of each sensitive word in the sensitive word list; the second determining unit is configured to determine the at least one candidate replacement word based on the sensitive word, the context words corresponding to the sensitive word, and at least one reference word in the reference word library.
[0092] In one embodiment of this disclosure, the first determining unit is specifically configured to: determine the part-of-speech type and / or sentiment type of the sensitive word; obtain a preset word library; select at least one word having the part-of-speech type and / or sentiment type from the preset word library according to the part-of-speech type and / or sentiment type of the sensitive word; and perform combination processing on the at least one word to obtain the reference word library.
[0093] In one embodiment of this disclosure, the second determining unit is specifically configured to: select at least one preliminary candidate replacement word from the at least one reference word based on the similarity between the sensitive word and the at least one reference word; determine the context adaptation score of the at least one preliminary candidate replacement word based on the context words corresponding to the sensitive word and the context adaptation scoring strategy; and select the at least one candidate replacement word from the at least one preliminary candidate replacement word based on the context adaptation score.
[0094] In one embodiment of this disclosure, the second determining unit is further configured to: determine the vector similarity between each preliminary candidate replacement word and each word in the context words for each preliminary candidate replacement word in the at least one preliminary candidate replacement word; and determine the mean of each vector similarity as the context adaptability score of the preliminary candidate replacement word.
[0095] In one embodiment of this disclosure, the second determining module 402 further includes: a third determining unit and a fourth determining unit; the third determining unit is configured to determine, for each of the at least one candidate replacement word, the parameter values of the candidate replacement word on at least two of the following parameters: the difference between 1 and the sensitivity probability of the candidate replacement word, the semantic similarity between the candidate replacement word and the sensitive word, and the context adaptability score of the candidate replacement word; the fourth determining unit is configured to determine the sensitivity protection index of the candidate replacement word based on the parameter values on the at least two parameters.
[0096] In one embodiment of this disclosure, the fourth determining unit is specifically used to determine the application scenario of the text and the weights of the at least two parameters under the application scenario; and to determine the sensitivity protection index of the candidate replacement word based on the at least two weights and the parameter values of the at least two parameters.
[0097] In one embodiment of this disclosure, the method for determining the sensitivity probability of the candidate replacement word includes: determining a first representation vector of the candidate replacement word; inputting the first representation vector into a word sense sensitivity model to obtain the sensitivity probability of the candidate replacement word.
[0098] In one embodiment of this disclosure, the first determining module 401 is specifically configured to: determine the text to be processed; perform word segmentation on the text to obtain a word segmentation list; perform vectorization on the word segmentation words in the word segmentation list to obtain a word segmentation vector list; input the word segmentation vector list into a word sense sensitivity model to obtain the sensitivity probability of each word segmentation word in the word segmentation list; and select sensitive words from each word segmentation word in the word segmentation list according to the sensitive word probability to obtain the sensitive word list.
[0099] In the text processing apparatus of this embodiment, the process involves determining the text to be processed and a list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word; for each sensitive word in the list, at least one candidate replacement word and a sensitivity protection index of the candidate replacement word are determined based on the sensitive word and its corresponding context words; a target replacement word corresponding to the sensitive word is selected from the at least one candidate replacement word based on the sensitivity protection index; and the text is processed by replacing sensitive words based on the target replacement word corresponding to at least one sensitive word in the text to obtain the processed text. By combining the target replacement word with the context words corresponding to the sensitive word and then performing the replacement processing, the process avoids missing text content, ensures easy understanding of the text content, avoids misunderstandings of the text content, and improves text processing efficiency.
[0100] According to a third aspect of the present disclosure, an electronic device is also provided, comprising: a processor; and a memory for storing processor-executable instructions, wherein the processor is configured to implement the text processing method described above.
[0101] To implement the above embodiments, this disclosure also proposes a storage medium.
[0102] When the instructions in the storage medium are executed by the processor, the processor is able to perform the text processing method described above.
[0103] To implement the above embodiments, this disclosure also provides a computer program product.
[0104] When the computer program product is executed by the processor of the electronic device, it enables the electronic device to perform the above-described method.
[0105] Figure 5 This is a structural block diagram of an electronic device according to an exemplary embodiment. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.
[0106] like Figure 5 As shown, the electronic device 1000 includes a processor 111, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 112 or a program loaded from memory 116 into random access memory (RAM) 113. The RAM 113 also stores various programs and data required for the operation of the electronic device 1000. The processor 111, ROM 112, and RAM 113 are interconnected via a bus 114. An input / output (I / O) interface 115 is also connected to the bus 114.
[0107] The following components are connected to I / O interface 115: memory 116 including hard disks, etc.; and communication section 117 including network interface cards such as local area network (LAN) cards, modems, etc., communication section 117 performs communication processing via a network such as the Internet; and driver 118 is also connected to I / O interface 115 as needed.
[0108] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 117. When the computer program is executed by processor 111, it performs the functions defined in the methods of this disclosure.
[0109] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, which can be executed by the processor 111 of the electronic device 1000 to perform the above-described method. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.
[0110] In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can transmit, propagate, or transfer a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.
[0111] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”
[0112] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”
[0113] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.
[0114] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A text processing method, characterized in that, The method includes: Determine the text to be processed and the list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word. For each sensitive word in the list of sensitive words, at least one candidate replacement word and a sensitivity protection index of the at least one candidate replacement word are determined based on the sensitive word and the context words corresponding to the sensitive word. Based on the sensitivity protection index, select the target replacement word corresponding to the sensitive word from the at least one candidate replacement word; Based on the target replacement word corresponding to at least one sensitive word in the text, the text is processed by sensitive word replacement to obtain the processed text.
2. The method according to claim 1, characterized in that, For each sensitive word in the list of sensitive words, at least one candidate replacement word is determined based on the sensitive word and its corresponding context words, including: For each sensitive word in the sensitive word list, a reference word library matching the part-of-speech type and / or sentiment type of the sensitive word is determined; The at least one candidate replacement word is determined based on the sensitive word, the context words corresponding to the sensitive word, and at least one reference word in the reference word library.
3. The method according to claim 2, characterized in that, For each sensitive word in the sensitive word list, determining a reference word library that matches the part-of-speech type and / or sentiment type of the sensitive word includes: Determine the part-of-speech type and / or sentiment type of the sensitive words; Obtain the preset word library; Based on the part-of-speech type and / or sentiment type of the sensitive words, at least one word having the part-of-speech type and / or sentiment type is selected from the preset word library; The at least one word is combined to obtain the reference word library.
4. The method according to claim 2, characterized in that, The step of determining the at least one candidate replacement word based on the sensitive word, the context words corresponding to the sensitive word, and at least one reference word in the reference word library includes: Based on the similarity between the sensitive words and the at least one reference word, at least one preliminary candidate replacement word is selected from the at least one reference word; Based on the context words corresponding to the sensitive words and the context adaptation scoring strategy, determine the context adaptation score of the at least one preliminary candidate replacement word; Based on the context adaptability score, at least one candidate replacement word is selected from at least one preliminary candidate replacement word.
5. The method according to claim 4, characterized in that, The step of determining the context adaptation score of the at least one preliminary candidate replacement word based on the context words corresponding to the sensitive words and the context adaptation scoring strategy includes: For each of the at least one preliminary candidate replacement words, determine the vector similarity between the preliminary candidate replacement word and each word in the context words; The mean of the similarity of each vector is determined as the context adaptability score of the preliminary candidate replacement word.
6. The method according to claim 1, characterized in that, For each sensitive word in the sensitive word list, a sensitivity protection index for the at least one candidate replacement word is determined based on the sensitive word and its corresponding context words, including: For each candidate replacement word in the at least one candidate replacement word, determine the parameter values of the candidate replacement word on at least two of the following parameters: the difference between 1 and the sensitivity probability of the candidate replacement word, the semantic similarity between the candidate replacement word and the sensitive word, and the contextual adaptability score of the candidate replacement word; The sensitivity protection index of the candidate replacement word is determined based on the parameter values of the at least two parameters.
7. The method according to claim 6, characterized in that, Determining the sensitivity protection index of the candidate replacement word based on the parameter values of the at least two parameters includes: Determine the application scenario of the text and the weights of at least two parameters in the application scenario; The sensitivity protection index of the candidate replacement word is determined based on at least two of the weights and the parameter values of the at least two parameters.
8. The method according to claim 6, characterized in that, The method for determining the sensitivity probability of the candidate replacement words includes: Determine the first representation vector of the candidate replacement word; Input the first representation vector into the word sense sensitivity model to obtain the sensitivity probability of the candidate replacement words.
9. The method according to claim 1, characterized in that, The determination of the text to be processed and the list of sensitive words corresponding to the text includes: Identify the text to be processed; The text is segmented into words to obtain a word list; The segmented words in the word segmentation list are vectorized to obtain a word segmentation vector list; Input the word segmentation vector list into the word sense sensitivity model to obtain the sensitivity probability of each word segmentation word in the word segmentation list; Based on the probability of the sensitive words, sensitive words are selected from each segmented word in the word segmentation list to obtain the sensitive word list.
10. A text processing device, characterized in that, The device includes: The first determining module is used to determine the text to be processed and the list of sensitive words corresponding to the text; the list of sensitive words includes at least one sensitive word. The second determining module is used to determine at least one candidate replacement word and a sensitivity protection index of the at least one candidate replacement word for each sensitive word in the sensitive word list, based on the sensitive word and the context words corresponding to the sensitive word. The selection module is used to select the target replacement word corresponding to the sensitive word from the at least one candidate replacement word according to the sensitivity protection index; The replacement processing module is used to perform sensitive word replacement processing on the text according to the target replacement word corresponding to at least one sensitive word in the text, so as to obtain the processed text.
11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured as follows: The steps of implementing the text processing method as described in any one of claims 1 to 9.
12. A non-transitory computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor, enable the processor to perform the text processing method as described in any one of claims 1 to 9.