Text processing method and device, equipment, storage medium and computer program product

Abstract

The embodiment of the invention discloses a text processing method. The method comprises the steps of generating a target statement based on a target statement structure feature; generating target dialogue information for the target statement based on a basic dialogue template and the target statement; wherein the target dialogue information is used for explaining the target statement; and embedding the target dialogue information into the basic sample text to obtain the to-be-verified text, thereby solving the problem that a watermark implantation method in related technologies can reduce the data set quality and influence the model capability, and expanding the applicable range of generating the protected data set. The embodiment of the invention further discloses a text processing device and equipment, a storage medium and a computer program product.

Description

Technical Field

[0001] This application relates to text processing technology in the field of communications, and more particularly to a text processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the emergence of Large Language Models (LLMs), text datasets have become a key resource driving progress in the field of Natural Language Processing (NLP). These datasets provide rich training resources for training large language models, significantly aiding in improving model performance and verifying algorithm feasibility. However, with the widespread use of text datasets, the protection of dataset copyright and intellectual property rights has become a problem that needs to be addressed. Once a dataset is used to train a large language model, its original data is difficult to extract directly from the trained model. Traditional digital rights management methods, such as directly embedding watermarks in the text, become inapplicable. Therefore, new copyright protection technologies need to be developed to ensure that data watermarks can still be verified after model training, thus protecting the rights of dataset copyright owners. Currently, methods for adding digital watermarks to datasets mainly include backdoor implantation, function implantation, and style implantation; however, backdoor implantation methods require modification of the dataset's annotations, which can cause discrepancies between the annotations of triggered samples and the actual annotations, thereby reducing the quality of the dataset. Meanwhile, the inserted trigger characters may be detected or affect the quality of the dataset; function implantation methods can only be applied to datasets containing mathematical functions, and watermarked functions with watermark embeddings used to generate explanatory dialogue text may interfere with the learning ability of large language models during training; style implantation methods can only be applied to datasets containing code. Therefore, watermark implantation methods in related technologies reduce dataset quality and affect model capabilities, and their application scope is relatively limited. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a text processing method, apparatus, device, storage medium, and computer program product, which solves the problem that watermarking methods in related technologies reduce dataset quality and affect model capabilities, and expands the applicability of generating protected datasets.

[0004] To achieve the above objectives, the technical solution of this application embodiment is implemented as follows:

[0005] A text processing method, the method comprising:

[0006] Generate the target statement based on its structural features;

[0007] Based on the basic dialogue template and the target statement, target dialogue information is generated for the target statement; wherein, the target dialogue information is used to interpret the target statement;

[0008] The target dialogue information is embedded into the base sample text to obtain the text to be verified.

[0009] In the above scheme, the step of generating the target statement based on the structural features of the target statement includes:

[0010] Based on the structural features of the target words, a basic sentence is generated, and the basic sentence is verified.

[0011] Upon successful verification, the target statement is determined based on the target uncommon character set and the basic statement.

[0012] In the above scheme, determining the target statement based on the target uncommon character set and the basic statement includes:

[0013] Based on the target information related to the character, obtain the matching rare characters that match the characters in the basic statement from the target rare character database;

[0014] The target statement is determined based on the matching of rare characters and the characters in the basic statement.

[0015] In the above scheme, generating target dialogue information for the target statement based on the basic dialogue template and the target statement includes:

[0016] Determine the basic dialogue template;

[0017] The basic dialogue template and the target statement are processed using a target language model to obtain basic dialogue information;

[0018] The statement logic of the basic dialogue information is processed to obtain the target dialogue information.

[0019] The method in the above scheme further includes:

[0020] The model to be verified is processed based on the target question information to determine whether the model to be verified uses the text to be verified.

[0021] In the above scheme, the step of processing the model to be verified based on the target question information to determine whether the model to be verified has used the text to be verified includes:

[0022] Determine the target question information corresponding to the target statement;

[0023] The target query information is processed using the model to be verified to obtain the processing result;

[0024] Based on the processing results, it is determined whether the model to be verified used the text to be verified.

[0025] In the above scheme, determining the target question information corresponding to the target statement includes:

[0026] Based on the target dialogue information, the target question information corresponding to the target statement is generated.

[0027] In the above scheme, determining whether the model to be verified uses the text to be verified based on the processing result includes:

[0028] The processing results are analyzed;

[0029] If the analysis results indicate that the processing results are correct, it is determined that the text to be verified was used in the model to be verified.

[0030] A text processing apparatus, the apparatus comprising:

[0031] The first generation unit is used to generate the target statement based on the structural features of the target statement.

[0032] The second generation unit is used to generate target dialogue information for the target statement based on the basic dialogue template and the target statement; wherein the target dialogue information is used to interpret the target statement;

[0033] The processing unit is used to embed the target dialogue information into the base sample text to obtain the text to be verified.

[0034] A text processing device, the device comprising: a processor, a memory, and a communication bus;

[0035] The communication bus is used to realize the communication connection between the processor and the memory;

[0036] The processor is used to execute the text processing program in the memory to implement the steps of the text processing method described above.

[0037] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the text processing method described above.

[0038] A computer program product includes a computer program that, when executed by a processor, implements the above-described text processing method.

[0039] The text processing method, apparatus, device, storage medium, and computer program product provided in the embodiments of this application can generate target sentences based on the structural features of target sentences, generate target dialogue information for target sentences based on basic dialogue templates and target sentences, and use the target dialogue information to interpret the target sentences. The target dialogue information is embedded in the basic sample text to obtain the text to be verified. In this way, the target dialogue information of the target sentences corresponding to the structural features of the target sentences can be generated, and the target dialogue information can be embedded in the basic sample text as watermark information, instead of using backdoor implantation, function implantation, and style implantation to determine the watermark information as in related technologies. This solves the problem that watermark implantation methods in related technologies reduce the quality of datasets and affect the capabilities of models, and expands the applicability of generating protected datasets. Attached Figure Description

[0040] Figure 1 A flowchart illustrating a text processing method provided for an embodiment of this application;

[0041] Figure 2 A flowchart illustrating another text processing method provided for an embodiment of this application;

[0042] Figure 3 A schematic diagram illustrating the process of generating target dialogue information and embedding basic sample text in a text processing method provided in an embodiment of this application;

[0043] Figure 4 A flowchart illustrating the process of verifying whether a model to be verified uses the text to be verified in a text processing method provided in an embodiment of this application;

[0044] Figure 5 A schematic diagram of the structure of a text processing device provided for an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the structure of a text processing device provided for an embodiment of this application. Detailed Implementation

[0046] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0047] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0048] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0049] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0050] This application provides a text processing method that can be applied to a text processing device. (Refer to...) Figure 1 As shown, the method may include the following steps:

[0051] Step 101: Generate the target statement based on the structural features of the target statement.

[0052] The target sentence structure features can refer to the typical structural features of a sentence that enable watermarking. In one feasible implementation, the target sentence structure features can refer to the sentence structure features corresponding to Chinese idioms, especially the sentence structure features corresponding to four-character Chinese idioms; the target sentence can refer to the target idiom.

[0053] In this embodiment of the application, the text processing device can use artificial intelligence technology to generate target sentences based on the structural features of the target sentences; specifically, artificial intelligence technology is used to generate target idioms (i.e., target sentences) that conform to the four-character format and have meaning based on the typical structural features of Chinese idioms (i.e., the structural features of target sentences).

[0054] Step 102: Based on the basic dialogue template and the target statement, generate target dialogue information for the target statement.

[0055] The target dialogue information is used to interpret the target statement.

[0056] In this embodiment, the basic dialogue template can be a pre-set dialogue template used to interpret the target statement, based on the actual application scenario and specific needs. The target dialogue information can be dialogue content that explains the target statement.

[0057] It should be noted that the text processing device can fill in the sentences based on the basic dialogue template and the target sentence to obtain the target dialogue information.

[0058] Step 103: Embed the target dialogue information into the basic sample text to obtain the text to be verified.

[0059] Here, the basic sample text can refer to the text dataset used for model training; the text to be verified can refer to the watermarked text dataset used for model training.

[0060] In this embodiment of the application, the text processing device can merge the target dialogue information into the basic sample text to obtain the text to be verified; specifically, the target dialogue information can be inserted into the basic sample text using a target embedding method.

[0061] In one feasible implementation, the target embedding method can be to insert the target dialogue information at a random position in the base sample text, or to insert the target dialogue information into the base sample text around the dialogue content of the target dialogue information.

[0062] The text processing method provided in the embodiments of this application can generate target dialogue information of the target sentence corresponding to the structural features of the target sentence, and embed the target dialogue information as watermark information into the basic sample text, instead of determining the watermark information as in related technologies. This solves the problem that watermark implantation methods in related technologies reduce the quality of the dataset and affect the model's capabilities, and expands the applicability of generating protected datasets.

[0063] Based on the foregoing embodiments, embodiments of this application provide a text processing method, referring to... Figure 2 As shown, the method may include the following steps:

[0064] Step 201: The text processing device generates basic sentences based on the structural features of the target words and verifies the basic sentences.

[0065] In the embodiments of this application, such as Figure 3The example shown illustrates how artificial intelligence (AI) technology can be used to generate basic sentences with the structural features of the target words. Alternatively, the structural features of the target words can be determined manually, and basic sentences can be generated based on these features. The basic sentences can refer to watermarked idioms, i.e., idioms with watermark functionality. Furthermore, text processing devices can verify the watermarked idioms to obtain the target idiom (i.e., the target sentence).

[0066] It should be noted that the text processing device can verify the watermarked idioms based on idioms in existing historical documents or related technologies; thus, it ensures that the underlying statements that pass the verification do not exist in existing historical documents or are not used in related technologies.

[0067] Step 202: Upon successful verification, the text processing device determines the target statement based on the target uncommon character set and the basic statement.

[0068] In this embodiment of the application, when the verification result indicates that the basic statement does not exist in existing historical documents or is not used in related technologies, it can be determined that the basic statement verification is successful. At this time, the text processing device can process the basic statement using a target uncommon character set to obtain the target statement. Specifically, uncommon characters can be inserted into the basic statement using the target uncommon character set to obtain the target statement.

[0069] In this embodiment of the application, the step 202 of "determining the target statement based on the target uncommon character set and basic statements" can be implemented in the following way:

[0070] Step 202a: The text processing device retrieves matching rare characters from the target rare character library that match the characters in the basic sentence, based on the target information related to the characters.

[0071] The target information can refer to parameters that uniquely distinguish each character; specifically, it can refer to parameters inherent in the character set application, i.e., character attribute parameters. The target rare character set can refer to a pre-constructed character set containing a large number of rare characters. In one feasible implementation, the target information can include parameters such as character shape and stroke count. The target rare character set can be constructed based on the "List of Commonly Used Characters in Modern Chinese" or other authoritative character lists.

[0072] In this embodiment, the text processing device can search for characters in a target rare character database that match the target information of characters in a basic statement to obtain the matching rare character. Specifically, each character in the basic statement can be compared with characters in the target rare character database based on factors such as stroke count and character shape. It should be noted that the stroke count and character shape of each character in the basic statement are compared with the stroke count and character shape of characters in the target rare character database. If there are characters in the target rare character database with the same stroke count and character shape as characters in the basic statement, then it is considered that there are characters in the target rare character database that match characters in the basic statement. At this time, characters with the same stroke count and character shape as characters in the target rare character database can be obtained from the target rare character database, thus obtaining the matching rare character.

[0073] Step 202b: The text processing device determines the target sentence based on matching rare characters and characters in the basic sentence.

[0074] In this embodiment, the target sentence can be obtained by modifying the characters in the base sentence using the matching uncommon characters. It should be noted that the target sentence can be obtained by modifying the characters in the base sentence that match the matching uncommon characters.

[0075] In one feasible implementation, for each rare character, the target statement can be obtained by replacing the characters in the base statement that match that rare character with the matching rare character.

[0076] In other embodiments of this application, if no matching character can be found for any of the characters in the basic statement in the target rare character set, then the basic statement will be discarded.

[0077] It should be noted that inserting uncommon characters into the basic sentences can help avoid interfering with the normal content of the basic sample text, thus protecting the basic sample text while ensuring that its content is not affected.

[0078] Step 203: The text processing device determines the basic dialogue template.

[0079] In this embodiment of the application, the basic dialogue template can be multiple predefined dialogue templates, used to demonstrate the meaning and usage scenarios of the target statement (i.e., the target idiom). Specifically, such as... Figure 3 As shown, the basic dialogue template can be generated by a text processing device using artificial intelligence and a large language model, resulting in several or dozens of predefined dialogue templates. Of course, the basic dialogue template can also be created by humans.

[0080] It should be noted that basic dialogue templates can include various different forms of dialogue templates. In one feasible implementation, basic dialogue templates can refer to dialogue templates in the form of situational dialogues, story dialogues, or question-and-answer sessions.

[0081] Step 204: The text processing device uses the target language model to process the basic dialogue template and target statement to obtain basic dialogue information.

[0082] The target language model can refer to a large language model. In the embodiments of this application, such as... Figure 3 The text processing device shown can use AI technology and a large language model to fill in the dialogue content of the basic dialogue template based on the basic dialogue template and the target sentence to obtain basic dialogue information.

[0083] Step 205: The text processing device processes the statement logic of the basic dialogue information to obtain the target dialogue information.

[0084] The target dialogue information is used to interpret the target statement.

[0085] In this embodiment of the application, the text processing device can correct the statement logic of the basic dialogue information to obtain accurate target dialogue information.

[0086] Specifically, such as Figure 3 The system can verify the statement logic of the basic dialogue information. If the statement logic is correct, the basic dialogue information is determined to be the final target dialogue information. If there is an error in the statement logic storage, the statement logic of the basic dialogue information is modified or the dialogue information is regenerated to obtain the target dialogue information.

[0087] It should be noted that the correctness of the statement logic of the basic dialogue information can be determined by judging whether the semantics of the basic dialogue information are fluent and whether it meets the requirements of the basic dialogue template.

[0088] Step 206: The text processing device embeds the target dialogue information into the basic sample text to obtain the text to be verified.

[0089] In the embodiments of this application, in such Figure 3 As shown, during the process of inserting target dialogue information into the base sample text, the embedding position and method of the target dialogue information in the base sample text can be recorded; then, the embedding position and method are stored confidentially.

[0090] In other embodiments of this application, during subsequent use of the text to be verified (i.e., the text dataset), a file recording the embedding locations can be sent to legitimate users. Thus, legitimate users can use the embedding locations to remove target dialogue information (i.e., watermark data) during use. It should be noted that when sending embedding location files to different legitimate users, different watermark idiom embedding locations can be retained, so as to serve as identity markers to determine which legitimate user is specifically using which watermark idiom. If a data leak subsequently occurs, the identity markers can be used to find the source of the leak.

[0091] In other embodiments of this application, the method may further include the following steps:

[0092] Step 207: The text processing device processes the model to be verified based on the target question information to determine whether the model to be verified has used the text to be verified.

[0093] In this embodiment, the target question information can be generated based on the target dialogue information and the target statement. Specifically, the text processing device can process the target question information using the model to be verified, and determine whether the model to be verified has used the text to be verified based on the processing result. It should be noted that before determining whether the model to be verified has used the text to be verified, the user permission for the model to be verified can be obtained first.

[0094] In one feasible implementation, such as Figure 4 The model to be verified could refer to one that is suspected of being trained using a protected text dataset, or it could refer to a large language model that needs to be verified to have used a protected text dataset.

[0095] It should be noted that step 207 can be achieved in the following way:

[0096] Step 207a: The text processing device determines the target query information corresponding to the target statement.

[0097] In this embodiment of the application, the target question information is used to ask questions to the model to be verified.

[0098] It should be noted that the target question information can include various forms of question content; in one feasible implementation, the target question information can include requesting the model to explain the meaning of the target idiom, providing example sentences or context related to the target idiom, or requiring the model to create content related to the target idiom.

[0099] In other embodiments of this application, step 207a can be implemented in the following ways:

[0100] a1. The text processing device generates target question information corresponding to the target statement based on the target dialogue information.

[0101] Specifically, the target question information can be obtained by generating question information for the target statement based on the dialogue characteristics of the target dialogue information.

[0102] Step 207b: The text processing device uses the model to be verified to process the target question information and obtain the processing result.

[0103] In the embodiments of this application, such as Figure 4 As shown, the target question information can be used to ask questions to the model to be validated, that is, the target question information is input into the model to be validated; then the model to be validated can analyze and answer the target question information to obtain the processing result. It should be noted that the processing result can include the model to be validated's answer to the target question information.

[0104] Step 207c: The text processing device determines whether the model to be verified has used the text to be verified based on the processing result.

[0105] In this embodiment of the application, the processing result can be analyzed to obtain the matching relationship between the processing result and the target answer, and then the model to be verified can be determined based on the matching relationship to determine whether the model to be verified uses the text to be verified.

[0106] It should be noted that step 207c can be achieved in the following way:

[0107] c1. The text processing device analyzes the processing results.

[0108] c2. If the analysis results represent the processing results correctly, the text processing device determines that the text to be verified was used in the model to be verified.

[0109] In this embodiment, the text processing device can compare the processing result with the target answer to obtain a matching relationship between the processing result and the target answer (i.e., obtain the analysis result). If the matching relationship is that the processing result is consistent with the target answer, the processing result can be considered correct, and the text processing device can determine that the model to be verified used the text to be verified during training. If the matching relationship is that the processing result is inconsistent with the target answer, the processing result can be considered incorrect, and the text processing device can determine that the model to be verified did not use the text to be verified during training.

[0110] The target answer can be pre-set based on the target question information and the target statement. It should be noted that the target answer can be set based on factors such as whether the model can correctly identify and interpret the target statement, whether the model's answer contains usages related to the target statement, and the consistency and stability of the model's answer. Furthermore, if the model to be verified can correctly identify and interpret the target statement without obvious errors or anomalies, it can be determined that the target statement is embedded in the model's training data, confirming that the model used protected text to be verified during training.

[0111] It should be noted that the text processing method provided in this application can generate watermarked idioms and explanatory dialogues from Chinese text, embed them as watermarks in the protected text dataset, and use the watermarked idioms as triggers. Furthermore, it can generate watermarked idioms that do not exist in existing records, and uses similar-looking but uncommon characters for replacement, avoiding interference with the performance of the trained large language model. In addition, the method for generating text to be verified provided in this application is applicable to any Chinese dataset and suitable for open-source dataset protection tasks in various fields. Compared to function implantation and style implantation methods in related technologies, it can be applied to all Chinese datasets. Moreover, the generated watermarked idioms do not exist in existing records and can be removed during normal use, without interfering with the performance of the trained large language model. Compared to traditional triggers, it does not degrade the performance of the dataset.

[0112] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0113] The text processing method provided in the embodiments of this application can generate target dialogue information of the target sentence corresponding to the structural features of the target sentence, and embed the target dialogue information as watermark information into the basic sample text, instead of determining the watermark information as in related technologies. This solves the problem that watermark implantation methods in related technologies reduce the quality of the dataset and affect the model's capabilities, and expands the applicability of generating protected datasets.

[0114] Based on the foregoing embodiments, embodiments of this application provide a text processing apparatus that can be applied to... Figure 1 and Figure 2 In the text processing method provided in the corresponding embodiment, refer to Figure 5 As shown, the text processing device 3 may include: a first generation unit 31, a second generation unit 32, and a processing unit 33, wherein:

[0115] The first generation unit 31 is used to generate a target statement based on the structural features of the target statement.

[0116] The second generation unit 32 is used to generate target dialogue information for the target statement based on the basic dialogue template and the target statement; wherein, the target dialogue information is used to interpret the target statement.

[0117] Processing unit 33 is used to embed target dialogue information into basic sample text to obtain text to be verified.

[0118] In other embodiments of this application, the first generation unit 31 is further configured to perform the following steps:

[0119] Generate basic sentences based on the structural features of target words, and validate the basic sentences;

[0120] If the verification passes, the target statement is determined based on the target uncommon character set and the basic statement.

[0121] In other embodiments of this application, the first generation unit 31 is further configured to perform the following steps:

[0122] Based on the target information related to the characters, obtain the matching rare characters that match the characters in the basic sentence from the target rare character database;

[0123] The target sentence is determined by matching rare characters with characters in the basic sentence.

[0124] In other embodiments of this application, the second generation unit 32 is further configured to perform the following steps:

[0125] Determine the basic dialogue template;

[0126] The basic dialogue template and target statements are processed using the target language model to obtain basic dialogue information;

[0127] The statement logic of the basic dialogue information is processed to obtain the target dialogue information.

[0128] In other embodiments of this application, the processing unit 33 is further configured to process the model to be verified based on the target question information to determine whether the model to be verified uses the text to be verified.

[0129] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps:

[0130] Determine the target query information corresponding to the target statement;

[0131] The target query information is processed using the model to be validated, and the processing result is obtained;

[0132] Based on the processing results, determine whether the model to be verified used the text to be verified.

[0133] In other embodiments of this application, the processing unit 33 is further configured to generate target question information corresponding to the target statement based on the target dialogue information.

[0134] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps:

[0135] Analyze the processing results;

[0136] If the analysis results correctly characterize the processing results, it is determined that the text to be verified was used in the model to be verified.

[0137] It should be noted that the specific implementation process of the steps performed by each unit in the embodiments of this application can be referred to Figure 1 and Figure 2 The implementation process of the text processing method provided in the corresponding embodiment will not be described in detail here.

[0138] The text processing apparatus provided in the embodiments of this application can generate target dialogue information of the target sentence corresponding to the structural features of the target sentence, and embed the target dialogue information as watermark information into the basic sample text, instead of determining the watermark information as in related technologies. This solves the problem that watermark implantation methods in related technologies reduce the quality of the dataset and affect the model's capabilities, and expands the scope of applicability.

[0139] Based on the foregoing embodiments, embodiments of this application provide a text processing device that can be applied to... Figure 1 and Figure 2 In the text processing method provided in the corresponding embodiment, refer to Figure 6 As shown, the text processing device 4 may include: a processor 41, a memory 42, and a communication bus 43, wherein:

[0140] Communication bus 43 is used to realize the communication connection between processor 41 and memory 42;

[0141] The processor 41 is used to execute the text processing program in the memory 42 to perform the following steps:

[0142] Generate the target statement based on its structural features;

[0143] Based on the basic dialogue template and the target statement, target dialogue information is generated for the target statement; the target dialogue information is used to interpret the target statement.

[0144] The target dialogue information is embedded into the base sample text to obtain the text to be verified.

[0145] In other embodiments of this application, processor 41 is used to execute a text processing program in memory 42 to generate a target statement based on the target statement's structural features, in order to implement the following steps:

[0146] Generate basic sentences based on the structural features of target words, and validate the basic sentences;

[0147] If the verification passes, the target statement is determined based on the target uncommon character set and the basic statement.

[0148] In other embodiments of this application, processor 41 is used to execute a text processing program in memory 42 to determine a target statement based on a target uncommon character set and basic statements, in order to implement the following steps:

[0149] Based on the target information related to the characters, obtain the matching rare characters that match the characters in the basic sentence from the target rare character database;

[0150] The target sentence is determined by matching rare characters with characters in the basic sentence.

[0151] In other embodiments of this application, processor 41 is used to execute a text processing program in memory 42 based on a basic dialogue template and a target statement to generate target dialogue information for the target statement, in order to implement the following steps:

[0152] Determine the basic dialogue template;

[0153] The basic dialogue template and target statements are processed using the target language model to obtain basic dialogue information;

[0154] The statement logic of the basic dialogue information is processed to obtain the target dialogue information.

[0155] In other embodiments of this application, the processor 41 is used to execute a text processing program in the memory 42, and may also perform the following steps:

[0156] The model to be validated is processed based on the target question information to determine whether the model has used the text to be validated.

[0157] In other embodiments of this application, processor 41 is used to execute a text processing program in memory 42 to process the model to be verified based on the target question information, and determine whether the model to be verified uses the text to be verified, in order to achieve the following steps:

[0158] Determine the target query information corresponding to the target statement;

[0159] The target query information is processed using the model to be validated, and the processing result is obtained;

[0160] Based on the processing results, determine whether the model to be verified used the text to be verified.

[0161] In other embodiments of this application, the processor 41 is used to execute the text processing program in the memory 42 to determine the target query information corresponding to the target statement, in order to implement the following steps:

[0162] Based on the target dialogue information, generate target question information corresponding to the target statement.

[0163] In other embodiments of this application, the processor 41 is used to execute a text processing program in the memory 42 to determine, based on the processing result, whether the model to be verified uses the text to be verified, in order to implement the following steps:

[0164] Analyze the processing results;

[0165] If the analysis results correctly characterize the processing results, it is determined that the text to be verified was used in the model to be verified.

[0166] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 and Figure 2 The text processing methods provided in the corresponding embodiments will not be described in detail here.

[0167] The text processing device provided in the embodiments of this application can generate target dialogue information of the target sentence corresponding to the structural features of the target sentence, and embed the target dialogue information as watermark information into the basic sample text, instead of determining the watermark information as in related technologies. This solves the problem that watermark implantation methods in related technologies reduce the quality of the dataset and affect the model's capabilities, and expands the scope of applicability.

[0168] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement... Figure 1 and Figure 2 The corresponding embodiments provide the steps of the text processing method.

[0169] Based on the foregoing embodiments, embodiments of this application provide a computer program product, including a computer program that can be executed by a processor 41 to perform... Figure 1 and Figure 2 The corresponding embodiments provide the steps of the text processing method.

[0170] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0171] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0172] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0173] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0174] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A text processing method, characterized in that, The method includes: Generate the target statement based on its structural features; Based on the basic dialogue template and the target statement, target dialogue information is generated for the target statement; wherein, the target dialogue information is used to interpret the target statement; The target dialogue information is embedded into the base sample text to obtain the text to be verified.

2. The method according to claim 1, characterized in that, The generation of the target statement based on the structural features of the target statement includes: Based on the structural features of the target words, a basic sentence is generated, and the basic sentence is verified. Upon successful verification, the target statement is determined based on the target uncommon character set and the basic statement.

3. The method according to claim 2, characterized in that, The process of determining the target statement based on the target rare character set and the basic statement includes: Based on the target information related to the character, obtain the matching rare characters that match the characters in the basic statement from the target rare character database; The target statement is determined based on the matching of rare characters and the characters in the basic statement.

4. The method according to claim 1, characterized in that, The step of generating target dialogue information based on the basic dialogue template and the target statement includes: Determine the basic dialogue template; The basic dialogue template and the target statement are processed using a target language model to obtain basic dialogue information; The statement logic of the basic dialogue information is processed to obtain the target dialogue information.

5. The method according to claim 1, characterized in that, The method further includes: The model to be verified is processed based on the target question information to determine whether the model to be verified uses the text to be verified.

6. The method according to claim 5, characterized in that, The process of processing the model to be verified based on the target question information to determine whether the model to be verified has used the text to be verified includes: Determine the target question information corresponding to the target statement; The target query information is processed using the model to be verified to obtain the processing result; Based on the processing results, it is determined whether the model to be verified used the text to be verified.

7. The method according to claim 6, characterized in that, Determining the target question information corresponding to the target statement includes: Based on the target dialogue information, the target question information corresponding to the target statement is generated.

8. The method according to claim 6, characterized in that, The step of determining whether the model to be verified used the text to be verified based on the processing result includes: The processing results are analyzed; If the analysis results indicate that the processing results are correct, it is determined that the text to be verified was used in the model to be verified.

9. A text processing device, characterized in that, The device includes: The first generation unit is used to generate the target statement based on the structural features of the target statement. The second generation unit is used to generate target dialogue information for the target statement based on the basic dialogue template and the target statement; wherein the target dialogue information is used to interpret the target statement; The processing unit is used to embed the target dialogue information into the base sample text to obtain the text to be verified.

10. A text processing device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute a text processing program in memory to implement the steps of the text processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the text processing method as described in any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the text processing method according to any one of claims 1 to 8.

Images