Text deduplication method and related device

Abstract

The present application discloses a text deduplication method and related devices. The text deduplication method includes: encoding each text in the current source computing node once to obtain the text encoding value of each text; dividing the text encoding value of each text into several types of encoding sets, wherein the similarity between the text encoding values ​​in the same type of encoding set is higher than the similarity between different text encoding values ​​in different types of encoding sets; for each type of encoding set, deduplicating the text corresponding to the encoding set in the current source computing node, wherein the text to be deduplicated is determined based on the similarity between the text encoding value of the text to be deduplicated and other texts in the encoding set. The above scheme can improve the deduplication efficiency.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a text deduplication method, a text deduplication device, a target communication system, an electronic device, and a computer-readable storage medium. Background Art

[0002] Text deduplication is a core technology in natural language processing (NLP) and data management. Its primary goal is to identify and eliminate duplicate text from a text collection. This technology is widely used in scenarios such as search engine optimization, data warehouse cleaning, copyright infringement detection, and preprocessing machine learning training datasets. With the explosive growth of internet data, efficient text deduplication algorithms are crucial for reducing storage costs, improving model training efficiency, and ensuring data quality.

[0003] The text deduplication method in the related art is to store the text in a database and use traditional database query, comparison, and deletion methods to achieve deduplication. This method has low deduplication efficiency. Summary of the Invention

[0004] The present application provides a text deduplication method, a text deduplication device, a target communication system, an electronic device and a computer-readable storage medium, which can solve the problem of low deduplication efficiency of text deduplication methods in related technologies.

[0005] The present application provides a text deduplication method, comprising: encoding each text in a current source computing node once to obtain a text encoding value of each text; dividing the text encoding value of each text into several categories of encoding sets, wherein the similarity between the text encoding values ​​in the same category of encoding sets is higher than the similarity between different text encoding values ​​between different categories of encoding sets; for each category of encoding sets, deduplicating the text corresponding to the encoding set in the current source computing node, wherein the text to be deduplicated is determined based on the similarity between the text to be deduplicated and the text encoding values ​​of other texts in the encoding set.

[0006] The present application provides a text deduplication method, comprising: receiving at least one type of coding set sent by at least one source computing node, where the coding set is obtained by the source computing node dividing the text coding values ​​of each text in the source computing node, and the text coding values ​​of each text are obtained by the source computing node encoding each text separately; combining the same type of coding sets sent by each source computing node; for each type of combined coding set, obtaining the similarity between the coding values ​​of each text in the combined coding set; based on the similarity, determining the text to be deduplicated in the combined coding set, and notifying the source computing node where the text to be deduplicated is located to delete the text to be deduplicated.

[0007] The present application provides a text deduplication device, comprising: an encoding module, a partitioning module, and a deduplication module. The encoding module is used to encode each text in the current source computing node once to obtain the text encoding value of each text; the partitioning module is used to divide the text encoding value of each text into several types of encoding sets, wherein the similarity between the text encoding values ​​in the same type of encoding set is higher than the similarity between different text encoding values ​​in different types of encoding sets; the deduplication module is used to dedupe the text corresponding to the encoding set in the current source computing node for each type of encoding set, wherein the text to be deduplicated is determined based on the similarity between the text encoding value of the text to be deduplicated and other texts in the encoding set.

[0008] The present application provides a text deduplication device, comprising: a receiving module, a combining module, an acquiring module, and a determining module. The receiving module is used to receive at least one type of coding set sent by at least one source computing node, where the coding set is obtained by the source computing node dividing the text coding values ​​of each text in the source computing node, where the text coding values ​​of each text are obtained by the source computing node encoding each text separately; the combining module is used to combine the same type of coding sets sent by each source computing node; the acquiring module is used to acquire the similarity between the coding values ​​of each text in the combined coding set for each type of coding set; and the determining module is used to determine the text to be deduplicated in the combined coding set based on the similarity, and to notify the source computing node where the text to be deduplicated is located to delete the text to be deduplicated.

[0009] The present application provides a target communication system, including several nodes, wherein the several nodes include at least one source computing node and at least one similarity computing node, and the source computing node and the similarity computing node are used in the above method.

[0010] The present application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above method.

[0011] The present application provides a computer-readable storage medium having program instructions stored thereon, which implement the above method when the program instructions are executed by a processor.

[0012] The above scheme encodes each text in the current source computing node once to obtain the text encoding value of each text; divides the text encoding value of each text into several categories of encoding sets, and the similarity between each text encoding value in the same category of encoding set is higher than the similarity between different text encoding values ​​between different categories of encoding sets; deduplication is performed on the text corresponding to the encoding set in each category of encoding set, and the text to be deduplicated is determined based on the similarity between the text to be deduplicated and the text encoding value of other texts in the encoding set. Therefore, the present application does not directly calculate the similarity of each text and deduplication based on the similarity of each text, but encodes each text into a text encoding value once and then divides it into several categories of encoding sets, and deduplication is performed based on the similarity between the text encoding values ​​in the encoding set in units of encoding sets, thereby reducing computing overhead and improving deduplication efficiency.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.

[0015] Figure 1 It is a schematic diagram of the structure of the target communication system provided by this application;

[0016] Figure 2 is a structural diagram of a specific example of the target communication system provided by this application;

[0017] Figure 3 is a structural diagram of another specific example of the target communication system provided by this application;

[0018] Figure 4 This is a flowchart of the first embodiment of the text deduplication method provided by this application;

[0019] Figure 5 This is a flowchart of the second embodiment of the text deduplication method provided by this application;

[0020] Figure 6 This is a flowchart of the third embodiment of the text deduplication method provided by this application;

[0021] Figure 7 This is a flowchart of the fourth embodiment of the text deduplication method provided by this application;

[0022] Figure 8 This is a schematic diagram of the code set allocation for each node in the target communication system of this application;

[0023] Figure 9This is another schematic diagram of the code set allocation for each node in the target communication system of this application;

[0024] Figure 10 This is another schematic diagram of the code set allocation for each node in the target communication system of this application;

[0025] Figure 11 This is a flowchart of the fifth embodiment of the text deduplication method provided by the present application;

[0026] Figure 12 This is a schematic diagram of the large model pre-training system based on PYTORCH in this application;

[0027] Figure 13 It is a schematic diagram of a specific example of the text deduplication method provided by this application;

[0028] Figure 14 is a diagram of the text encoding value of the text of this application;

[0029] Figure 15 This is a schematic diagram of the coding set obtained by dividing the four nodes of this application;

[0030] Figure 16 It is a comparison diagram of the Jaccard similarity between the text encoding values ​​of the text and the true similarity between the texts;

[0031] Figure 17 This is a structural diagram of an embodiment of a text deduplication device of the present application;

[0032] Figure 18 It is a structural diagram of another embodiment of the text deduplication device of the present application;

[0033] Figure 19 This is a structural diagram of an embodiment of an electronic device of the present application;

[0034] Figure 20 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0035] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0036] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0037] The term "and / or" in this article is simply a description of the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0038] Figure 1 This is a schematic diagram of the structure of the target communication system provided by this application. Figure 1 As shown, the target communication system includes several nodes. Each node has text deduplication capabilities and can be a physical machine, virtual machine, or container instance. A physical machine can be a computer, mobile phone, server, or other electronic device with text deduplication computing capabilities. Each node can communicate with each other.

[0039] The plurality of nodes include at least one source computing node and at least one similarity computing node. The same node can be both a source computing node and a similarity computing node.

[0040] Figure 2 This is a schematic diagram of a specific example of the target communication system provided by this application. Figure 2 As shown, each node in the plurality of nodes is both a source computing node and a similarity computing node. For example, the plurality of nodes includes a node, which is both a source computing node and a similarity computing node. For another example, the plurality of nodes includes nodes A through D, which are both source computing nodes and similarity computing nodes.

[0041] Figure 3 is a structural diagram of another specific example of the target communication system provided by this application, such as Figure 3 As shown, some of the nodes are source computing nodes, and the other nodes are similarity computing nodes. For example, the nodes include nodes A to D, nodes A to B are source computing nodes, and nodes C to D are similarity computing nodes.

[0042] The source computing node stores text. In some embodiments, the plurality of nodes includes one source computing node, in which case all text is stored in the source computing node. In some embodiments, the plurality of nodes may also include multiple source computing nodes, in which case all text is evenly or unevenly distributed and stored in each source computing node.

[0043] The target communication system can be a large model pre-training system based on PYTORCH, a specially built communication system, or other existing communication systems.

[0044] The text deduplication method provided by this application can be implemented based on the above-mentioned target communication system. The following describes an embodiment of the text deduplication method provided by this application. It should be noted that each source computing node has a similar text deduplication process for its stored text. In the following embodiments of this application, only one source computing node (the current source computing node) is used as an example to introduce the text deduplication method provided by this application.

[0045] Figure 4 This is a flow chart of the first embodiment of the text deduplication method provided by this application. Figure 4 As shown, in this embodiment, the text deduplication method may include the following steps:

[0046] S110: Encode each text in the current source computing node once to obtain a text encoding value of each text.

[0047] The execution subject of this embodiment is the current source computing node.

[0048] In some embodiments, from a content perspective, text can be comments, opinions, conversations, logs, news, etc. The text in the current source computing node refers to the text stored in the current source computing node. From an application scenario perspective, text can be pre-trained text for a large model, text to be queried in a database, etc.

[0049] In some embodiments, the one-shot encoding method may be a hash encoding method, such as Minhash or simHash. Alternatively, the one-shot encoding method may be a statistical encoding method, such as a bag-of-words model or TF-IDF. Alternatively, the one-shot encoding method may be a word embedding-based encoding method, such as a static word vector or a context-dependent word vector. Alternatively, the one-shot encoding method may be a semantic encoding method based on deep learning, etc.

[0050] In some embodiments, each text may be directly encoded once to obtain a text encoding value of each text.

[0051] In some embodiments, each text may be divided into a number of text segments and then encoded once. In this case, the text encoding value of the text includes the encoding sub-values ​​of each text segment.

[0052] In some embodiments, the text encoding value of the text also includes location information of the text. The location information of the text includes first address information of the current source computing node where the text is located, and / or second address information of the text at the current source computing node. The first address information is used to identify the location of the current source computing node where the text is located, and the second address information is used to identify the storage location of the text at the current source computing node.

[0053] S120: Divide the text encoding value of each text into several types of encoding sets.

[0054] Among them, the similarity between text encoding values ​​within the same type of encoding set is higher than the similarity between different text encoding values ​​between different types of encoding sets.

[0055] The similarity between text encoding values ​​can represent the similarity between corresponding texts. Moreover, the similarity between text encoding values ​​is positively correlated with the similarity between corresponding texts. Specifically, the higher the similarity between text encoding values, the higher the similarity between corresponding texts. Conversely, the lower the similarity between text encoding values, the lower the similarity between corresponding texts. For example, the higher the similarity between the text encoding value a of text 1 and the text encoding value b of text 2, the higher the similarity between text 1 and text 2.

[0056] In some embodiments, the same text encoding values ​​may be classified into the same type of encoding set, and different text encoding values ​​may be classified into different types of encoding sets.

[0057] In some embodiments, the text code values ​​of each text may be clustered based on the similarity between the text code values ​​of each text to obtain a plurality of class code sets. Thus, text code values ​​with a similarity greater than or equal to a similarity threshold are grouped into the same class code set, and text code values ​​with a similarity less than the similarity threshold are grouped into different class code sets.

[0058] It can be understood that, compared with the method of directly classifying different text encoding values ​​into different types of encoding sets, similarity-based clustering can reduce the number of categories in the encoding set.

[0059] In some embodiments, S120 includes S121-S122. S121: For each text, the text encoding value of the text is secondary encoded to obtain the category encoding value of the text. S122: The text encoding values ​​corresponding to the texts whose category encoding values ​​have a difference degree less than a preset difference degree are divided into the same encoding set. The secondary encoding method can be hash encoding or other encoding methods. The secondary encoding method can be the same as or different from the primary encoding method. The difference degree less than the preset difference degree can be divided into two cases. One case is that the difference degree is equal to 0, that is, the category encoding values ​​are the same. The other case is that the difference degree is greater than 0 and less than the preset difference degree, that is, the category encoding values ​​are similar.

[0060] In some embodiments, S122 includes: dividing the text coding values ​​corresponding to texts with the same category coding value into the same category coding set.

[0061] It can be understood that compared with the method of directly classifying the text code values ​​corresponding to texts with the same category code values ​​into different category code sets, the method of classifying the text code values ​​corresponding to texts with the same and similar category code values ​​into different category code sets can reduce the number of categories in the code set.

[0062] S130: For each type of code set, deduplicate the text corresponding to the code set in the current source computing node.

[0063] The text to be deduplicated is determined based on the similarity between the text to be deduplicated and the text encoding values ​​of other texts in the encoding set.

[0064] In some embodiments, the texts that need to be deduplicated corresponding to each type of encoding set are determined by the current source computing node.

[0065] In some embodiments, the text to be deduplicated corresponding to at least one type of code set is determined by the current source computing node. The text to be deduplicated corresponding to other types of code sets is determined by other source computing nodes / similarity computing nodes. In the case where the text to be deduplicated corresponding to the same type of code set is determined by the same source computing node / similarity computing node, the fewer the number of categories of the code set, the fewer the number of source computing nodes / similarity computing nodes required to determine the text to be deduplicated.

[0066] In some embodiments, the to-be-deduplicated texts corresponding to various encoding sets are determined by other source computing nodes / similarity computing nodes.

[0067] The above scheme encodes each text in the current source computing node once to obtain the text encoding value of each text; divides the text encoding value of each text into several categories of encoding sets, and the similarity between each text encoding value in the same category of encoding set is higher than the similarity between different text encoding values ​​between different categories of encoding sets; deduplication is performed on the text corresponding to the encoding set in each category of encoding set, and the text to be deduplicated is determined based on the similarity between the text to be deduplicated and the text encoding value of other texts in the encoding set. Therefore, the present application does not directly calculate the similarity of each text and deduplication based on the similarity of each text, but encodes each text into a text encoding value once and then divides it into several categories of encoding sets, and deduplication is performed based on the similarity between the text encoding values ​​in the encoding set in units of encoding sets, thereby reducing computing overhead and improving deduplication efficiency.

[0068] Figure 5 This is a flow chart of the second embodiment of the text deduplication method provided by this application. This embodiment is a further extension of S110. In this embodiment, the text is divided into several text segments and then encoded again. Figure 5 As shown, in this embodiment, S110 may include the following steps:

[0069] S210: For each text, divide the text into several text segments.

[0070] In some embodiments, the text may be divided into several text segments based on units such as paragraphs, sentences, words, and characters.

[0071] In some embodiments, the text may be divided into several text segments using methods such as NGram and sliding windows.

[0072] S220: Encode each text segment of the text once to obtain the encoding sub-value of each text segment of the text.

[0073] S230: Obtain the text encoding value of the text using the encoding sub-values ​​of each text segment of the text.

[0074] In some embodiments, the encoding sub-values ​​of each text segment of the text can be combined to obtain the text encoding value of the text.

[0075] In some embodiments, the encoding sub-values ​​of each text segment of the text and the text location information can be combined to obtain the text encoding value of the text. The text location information includes the first address information of the current source computing node where the text is located and the second address information of the text at the current source computing node. Alternatively, the text location information only includes the first address information of the current source computing node where the text is located.

[0076] It is understandable that when the text encoding value of the text includes the first address information and the second address information of the text, it supports sending the encoding set to other source computing nodes / similarity computing nodes to determine the text to be deduplicated through other source computing nodes / similarity computing nodes.

[0077] Unlike the previous embodiment, this embodiment divides each text into several segments and then performs a single encoding, which can improve the accuracy of the text encoding value in representing the text. Furthermore, the text encoding value can carry the text's location information, supporting the sending of the encoding set to other source computing nodes / similarity computing nodes to determine the text to be deduplicated.

[0078] Furthermore, in some embodiments, the communication system where the current source computing node is located is the target communication system, and several source computing nodes in the target communication system each contain several texts, and the current source computing node is one of the several source computing nodes. Based on this, S130 can be expanded as follows:

[0079] In S130, at least one of the plurality of code sets may be used as a first code set, and / or at least one of the plurality of code sets may be used as a second code set. The text to be deduplicated corresponding to the first code set is determined by the current source computing node, and the text to be deduplicated corresponding to the second code set is determined by other source computing nodes / similar computing nodes.

[0080] In some embodiments, all the class code sets in the several class code sets are used as the first class code sets.

[0081] In some embodiments, among several types of coding sets, one type of coding set is used as a first type of coding set, and the other types of coding sets are used as a second type of coding sets.

[0082] In some embodiments, among several types of code sets, a first predetermined number of code sets are used as a first type of code set, and the remaining types of code sets are used as a second type of code set. The first predetermined number is greater than 1 and less than the total number of categories in the code set. For example, if there are 4 categories in the code set, the first predetermined number may be 2 or 3.

[0083] In some embodiments, among several types of code sets, a second preset number of code sets are used as the second type of code set, and the other types of code sets are used as the first type of code set. The second preset number is greater than 1 and less than the total number of categories in the code set. For example, if there are 4 categories in the code set, the second preset number can be 2 or 3.

[0084] In some embodiments, all the class code sets in the several class code sets are used as the second class code sets.

[0085] It is understandable that, when at least one type of coding set is used as the second type of coding set, the second type of coding set needs to be sent to other source computing nodes / similar computing nodes, so that the other source computing nodes / similar computing nodes can determine the text that needs to be deduplicated corresponding to the second type of coding set, and receive the relevant data (second address information or text coding value) of the text that needs to be deduplicated determined by the other source computing nodes / similar computing nodes. Since the amount of data of the second type of coding set / text coding value / second address information is smaller than the text, compared to directly transmitting the text between the current source computing node and other source computing nodes / similar computing nodes, the present application only transmits the second type of coding set / text coding value / second address information between nodes. The text will always be stored only in the source computing node and will not be transmitted between the source computing nodes. This can greatly reduce the amount of data transmitted between nodes, thereby improving communication efficiency and further improving deduplication efficiency. In addition, the number of times a node reads and writes the text it stores is reduced, and the storage space occupied by the node by the transmitted data is reduced.

[0086] Figure 6 This is a flow chart of the third embodiment of the text deduplication method provided by this application. This embodiment is a further extension of S130. Figure 6 As shown, in this embodiment, S130 may include the following steps:

[0087] S310: Use at least one type of coding set as a first type of coding set.

[0088] In some embodiments, all types of code sets may be used as first type code sets.

[0089] In some embodiments, a first preset number of code sets may be used as the first type of code set.

[0090] In some embodiments, one of the coding sets may be used as the first coding set.

[0091] S320: For each first-category code set, combine the first-category code sets in each source computing node, and calculate the similarity between each text code value in the combined first-category code set.

[0092] The first type code set in each source computing node includes the first type code set retained by the current source computing node and the first type code set received by the current source computing node and sent by other source computing nodes.

[0093] In some embodiments, the first category code set in each source computing node is the set of all category codes obtained by the corresponding source computing node. In this case, it can be considered that all category code sets obtained by the source computing node are aggregated to the current source computing node, and the current source computing node determines the text to be deduplicated.

[0094] In some embodiments, the first set of codes in each source computing node is a set of codes of the first predetermined number of categories obtained by the corresponding source computing node. In this case, the first set of codes of the first predetermined number of categories obtained by the source computing node can be considered to be aggregated to the current source computing node, and the current source computing node determines the text to be deduplicated.

[0095] In some embodiments, the first type of code set in each source computing node is one of the code sets obtained by the corresponding source computing node. In this case, it can be considered that the code sets obtained by the division of each source computing node are aggregated into the current source computing node, and the current source computing node determines the text to be deduplicated. In this case, it can be considered that the code sets of the same type obtained by the division of each source computing node are aggregated into the current source computing node, and the current source computing node determines the text to be deduplicated.

[0096] In some embodiments, when the first type coding set of each source computing node includes only one type of coding set, the coding sets of each source computing node may be combined to obtain a combination result of the first type coding set.

[0097] In some embodiments, when the first-category coding set of each source computing node includes multiple coding sets, each type of first-category coding set of each source computing node can be combined to obtain a combination result of each type of first-category coding set. For example, if the first-category coding set of each source computing node includes a coding set of category 1 and a coding set of category 2, the coding sets of category 1 of each source computing node can be combined to obtain a combination result of category 1, and the coding sets of category 2 of each source computing node can be combined to obtain a combination result of category 2.

[0098] S330: The texts corresponding to the text encoding values ​​whose similarities meet the similarity requirements in the combined first type encoding set are taken as repeated text groups.

[0099] In some embodiments, the similarity requirement can be satisfied when the similarity between any two text encoding values ​​is greater than a similarity threshold. For example, if there are 100 texts in the combined first-category encoding set, the similarity between any two of "50 of these texts" is greater than or equal to the similarity threshold, and the similarity between any two of "the other 50 texts" is less than the similarity threshold, then the "50 of these texts" are considered a duplicate text group.

[0100] In some embodiments, the similarity that satisfies the similarity requirement may also be the similarity between two of the encoding values ​​of some texts, and the difference between the similarity between two of the encoding values ​​of other texts is greater than a difference threshold. For example, there are 100 texts in the combined first-category encoding set, and the similarity between two of "50 of them" is greater than a first similarity, and the similarity between two of "the other 50 texts" is less than a second similarity, the first similarity is greater than the second similarity, and the difference between the second similarity and the first similarity is greater than the difference threshold, and the "50 of them" are treated as a repeated text group.

[0101] S340: Select at least one text from the duplicate text group as the first text to be deduplicated, delete the first text to be deduplicated located at the current source computing node, and notify the source computing node where the non-local text is located to delete the non-local text.

[0102] The non-local text is the first text to be deduplicated that is not located at the current source computing node.

[0103] Each text in the duplicate text group is a duplicate text. For duplicate text, only one needs to be retained in each source computing node. Therefore, at least one text needs to be selected from the duplicate text group as the first text to be deduplicated and deleted in the corresponding source computing node.

[0104] Since the duplicate text group is determined based on the first-category coding set retained by the current source computing node and the received first-category coding set, the duplicate text group may contain the first text to be deduplicated corresponding to the retained first-category coding set, and may also contain the first text to be deduplicated corresponding to the received first-category coding set. The first text to be deduplicated corresponding to the retained first-category coding set is stored in the current source computing node and is a local text. The first text to be deduplicated corresponding to the received first-category coding set is not stored in the current source computing node, but is stored in other source computing nodes and is a non-local text. Therefore, local text can be deleted directly, and non-local text needs to be notified to the source computing node where it is located to delete it.

[0105] Different from the above-mentioned embodiment, in this embodiment, the first category of coding sets obtained by each source computing node is summarized to the current source computing node, so as to determine the first text to be deduplicated corresponding to the first category of coding set through the current source computing node, thereby saving computing overhead.

[0106] Figure 7 This is a flowchart of the fourth embodiment of the text deduplication method provided by this application. This embodiment is a further extension of S130. Figure 7 As shown, in this embodiment, S130 may include the following steps:

[0107] S410: Using at least one type of coding set as a second type of coding set.

[0108] In some embodiments, all types of code sets may be used as the second type of code sets.

[0109] In some embodiments, a second preset number of code sets may be used as the second type of code set.

[0110] In some embodiments, one of the coding sets may be used as the second coding set.

[0111] S420: For each second-category coding set, the second-category coding set of the current source computing node is sent to the similarity computing node in the target communication system, so that the similarity computing node combines the second-category coding sets of each source computing node, and based on the similarity between the text coding values ​​in the combined second-category coding set, determines the second text to be deduplicated corresponding to the combined second-category coding set, and notifies the source computing node where the second text to be deduplicated is located to delete the second text to be deduplicated.

[0112] In some embodiments, the similarity calculation node is a source calculation node. Alternatively, the similarity calculation node is not a source calculation node.

[0113] In some embodiments, if there are multiple second-category code sets, each second-category code set of the current source computing node can be sent to a different similarity computing node. Alternatively, some second-category code sets of the current source computing node can be sent to the same similarity computing node. Alternatively, all second-category code sets of the current source computing node can be sent to the same similarity computing node.

[0114] In some embodiments, when the similarity calculation node is not a source calculation node, the second-category code set of each source calculation node includes the second-category code set sent by each source calculation node and received by the similarity calculation node. The similarity calculation node may combine the second-category code sets of each source calculation node to obtain a combined second-category code set; based on the similarity between the text code values ​​in the combined second-category code set, determine the second to-be-deduplicated text (non-local) corresponding to the combined second-category code set, and notify the source calculation node to delete the second to-be-deduplicated text.

[0115] In some embodiments, when the similarity calculation node is a source calculation node, the second-category code set of each source calculation node includes the second-category code set retained by the similarity calculation node and the second-category code sets sent by other source calculation nodes (including the current source calculation node). The similarity calculation node may combine the second-category code sets of each source calculation node to obtain a combined second-category code set; based on the similarity between the text code values ​​in the combined second-category code set, determine the second text to be deduplicated (local and non-local) corresponding to the combined second-category code set; directly delete the local second text to be deduplicated; and notify the source calculation node of the non-local second text to be deduplicated to delete it.

[0116] In some embodiments, when the second type code set of each source computing node includes only one type of code set, the similarity computing node may combine the code sets of this type of source computing node to obtain a combination result of the second type code set.

[0117] In some embodiments, when the second-category code set of each source computing node includes multiple code sets, the similarity calculation node may combine each type of second-category code set of each source computing node to obtain a combination result of each type of second-category code set. For example, if the second-category code set of each source computing node includes a code set of category 3 and a code set of category 4, the code sets of category 3 of each source computing node may be combined to obtain a combination result of category 3, and the code sets of category 4 of each source computing node may be combined to obtain a combination result of category 4.

[0118] Different from the above embodiment, in this embodiment, the second type of code set obtained by each source computing node is aggregated into the same similarity computing node, so as to determine the second text to be deduplicated through the similarity computing node, thereby saving computing overhead.

[0119] To facilitate understanding of S310 - S340 and S410 - S420 , the determination of the text to be deduplicated (the first text to be deduplicated and the second text to be deduplicated) is described below in the form of three specific examples.

[0120] Example 1: All class code sets obtained by the current source computing node are used as the first class code set. The first class code sets of each source computing node are aggregated to the current source computing node, and the current source computing node determines the first text to be deduplicated.

[0121] Figure 8 This is a schematic diagram of the code set allocation of each node in the target communication system of this application. Figure 8As shown, the target communication system includes nodes AD, which are both source and similarity computation nodes. Node A is the current source computation node. Node A divides the code sets A1-A4 into categories 1-4, node B divides the code sets B1-B4 into categories 1-4, node C divides the code sets C1-C4 into categories 1-4, and node D divides the code sets D1-D4 into categories 1-4. Nodes AD select A1-A4, B1-B4, C1-C4, and D1-D4 as the first-category code sets. Node BD sends the four code sets B1-B4, C1-C4, and D1-D4 to node A, respectively. Node A combines the coding sets A1, B1, C1, and D1 of category 1 to obtain the combination results of category 1, combines the coding sets A2, B2, C2, and D2 of category 2 to obtain the combination results of category 2, combines the coding sets A3, B3, C3, and D3 of category 3 to obtain the combination results of category 3, and combines the coding sets A4, B4, C4, and D4 of category 4 to obtain the combination results of category 4; the first text to be deduplicated is determined based on the combination results of categories 1-4 respectively.

[0122] Example 2: One of the code sets divided by the current source computing node is used as the first code set, and the other code sets are used as the second code set. The first code sets of each source computing node are aggregated to the current source computing node, which determines the first text to be deduplicated. The second code sets of each source computing node are aggregated to the similarity computing node, which determines the second text to be deduplicated.

[0123] Figure 9 This is another schematic diagram of the code set allocation of each node in the target communication system of this application. Figure 8 Yes, in Figure 9 In the code set, nodes AD respectively use the coding sets A1, B1, C1, and D1 of category 1 as the first category coding sets, and A2-A4, B2-B4, and C2-C4 as the second category coding sets. The first category coding sets A1, B1, C1, and D1 are aggregated to node A, and node A determines the corresponding first text to be deduplicated. The coding sets A2, B2, C2, and D2 of category 2 in the second category coding set are aggregated to node B, and node B determines the corresponding second text to be deduplicated. The coding sets A3, B3, C3, and D3 of category 3 in the second category coding set are aggregated to node C, and node C determines the corresponding second text to be deduplicated. The coding sets A4, B4, C4, and D4 of category 4 in the second category coding set are aggregated to node D, and node D determines the corresponding second text to be deduplicated.

[0124] Example 3: All class code sets obtained by the current source computing node are used as the second class code set, and the first class code sets of each source computing node are summarized to the similarity computing node, which determines the second text to be deduplicated.

[0125] It is understandable that in Example 2 and Example 3 and similar cases, the fewer the number of categories in the encoding set, the fewer the number of source calculation nodes / similarity calculation nodes required to determine the text to be deduplicated.

[0126] Figure 10 This is another schematic diagram of the code set allocation of each node in the target communication system of this application. Figure 8 and Figure 9 ,exist Figure 10 In the embodiment, the target communication system further includes a node EF, and the node EF is a similarity calculation node.

[0127] Nodes AD respectively use the code sets A1-A4, B1-B4, C1-C4, and D1-D4 of categories 1-4 as the second code set. The code sets A1-A2, B1-B2, C1-C2, and D1-D2 of categories 1-2 in the second code set are aggregated to node E. Node E combines the code sets A1, B1, C1, and D1 of category 1 to obtain the combination result of category 1, and combines the code sets A2, B2, C2, and D2 of category 2 to obtain the combination result of category 2. Based on the combination results of category 1 and category 2, the corresponding second text to be deduplicated is determined. The coding sets A3-A4, B3-B4, C3-C4, and D3-D4 of categories 3-4 in the second category of coding sets are summarized to node F. Node F combines the coding sets A3, B3, C3, and D3 of category 3 to obtain the combination result of category 3, and combines the coding sets A4, B4, C4, and D4 of category 4 to obtain the combination result of category 4. The corresponding second text to be deduplicated is determined based on the combination results of category 3 and category 4, respectively.

[0128] Further, in some embodiments, in S340, the source computing node where the non-local text is located is notified to delete the non-local text, or in S420, the source computing node where the second text to be deduplicated is located is notified to delete the second text to be deduplicated, including S510-S530 (not shown). S510: Use the non-local text or the second text to be deduplicated as the target deduplicated text. S520: From the text encoding value of the target deduplicated text, obtain the first address information of the source computing node where the target deduplicated text is located and the second address information of the target deduplicated text in the source computing node. S530: According to the first address information, the second address information is sent to the source computing node where the target deduplicated text is located to notify the source computing node to delete the text corresponding to the second address information.

[0129] In some embodiments, different from S530, the text encoding value may be sent to the source computing node where the target deduplicated text is located to notify the source computing node to delete the text corresponding to the second address information.

[0130] In some embodiments, the text deduplication method provided herein is executed by multiple nodes of a distributed target communication system. If the similarity calculation nodes are all source calculation nodes, then the multiple nodes are multiple source calculation nodes. If the similarity calculation nodes are not source calculation nodes, then the multiple nodes may include multiple source calculation nodes and similarity calculation nodes.

[0131] Based on this, the text deduplication method also includes: when data needs to be sent to other nodes, converting the data to be sent into a format supported by the target communication system. The distributed target communication system can be a large model pre-training system based on Pytorch, Hadoop, Spark, etc.

[0132] For example, the text deduplication method provided in this application is executed by several nodes under the large model pre-training system based on PYTORCH, and the several nodes include the current source computing node. Based on this, the text deduplication method also includes: when data needs to be sent to other nodes, converting the data to be sent into a format supported by the large model pre-training system based on PYTORCH.

[0133] The data to be sent can be a code set, the second address information of the target deduplicated text in the source computing node, the text encoding result, etc. Specifically, after dividing into several categories of code sets, if the code set needs to be sent, the code set is converted into a format supported by the large model pre-training system based on PYTORCH before sending. After determining the target deduplicated text, the second address information of the target deduplicated text in the source computing node or the text encoding result is converted into a format supported by the large model pre-training system based on PYTORCH before sending.

[0134] It's understandable that the PYTORCH-based large-model pre-training system is a communication system for implementing large-model pre-training. When the text deduplication method is executed by multiple nodes within the PYTORCH-based large-model pre-training system, there's no need to build a separate target communication system, reducing the cost of setting up the target communication system. Furthermore, the PYTORCH-based large-model pre-training system has built-in efficient distributed communication algorithms and various communication protocols. Data processing can be performed using both CPUs and GPUs (across multiple nodes). Data communication is stable, and communication strategies can be independently designed using built-in P2P interfaces. PYTORCH's underlying architecture uses C++ to optimize algorithm efficiency, and serialization schemes can be customized. It's compatible with any complex data structure and can freely scale the number of nodes. This results in high communication efficiency, stability, and compatibility.

[0135] Figure 11 This is a flowchart of the fifth embodiment of the text deduplication method provided by this application. Figure 11 As shown, in this embodiment, the text deduplication method may include the following steps:

[0136] S610: Receive at least one type of coding set sent by at least one source computing node.

[0137] The code set is obtained by dividing the text code values ​​of each text in the source computing node by the source computing node, and the text code value of each text is obtained by encoding each text by the source computing node respectively.

[0138] The execution subject of this embodiment is the similarity calculation node.

[0139] S620: Combine the same type of code sets sent by each source computing node.

[0140] For example, the coding sets of category 1 and category 2 sent by each source computing node are received, the coding sets of category 1 sent by each source computing node are combined to obtain the combination result of category 1, and the coding sets of category 2 sent by each source computing node are combined to obtain the combination result of category 2.

[0141] S630: For each type of combined code set, obtain the similarity between each text code value in the combined code set.

[0142] For example, the similarity between each text encoding value in the combination result of category 1 is obtained, and the similarity between each text encoding value in the combination result of category 2 is obtained.

[0143] S640: Based on the similarity, determine the text to be deduplicated in the combined encoding set, and notify the source computing node where the text to be deduplicated is located to delete the text to be deduplicated.

[0144] In some embodiments, notifying the source computing node where the text to be deduplicated is located to delete the text to be deduplicated includes: obtaining first address information of the source computing node where the text to be deduplicated is located and second address information of the text to be deduplicated in the source computing node from the text encoding value of the text to be deduplicated; and sending the second address information to the source computing node where the text to be deduplicated is located according to the first address information, so as to notify the source computing node to delete the text corresponding to the second address information.

[0145] For other detailed descriptions related to this embodiment, please refer to the previous embodiments and will not be repeated here.

[0146] Unlike the previous embodiment, this embodiment aggregates and combines the same code sets from each source computing node to obtain combined code sets. The text to be deduplicated is then determined based on the similarity between the code values ​​of the texts in the combined code sets. This reduces computational overhead.

[0147] For ease of understanding, the text deduplication method provided by this application is described below in the form of a specific example:

[0148] See also Figure 12 , Figure 12 This is a schematic diagram of the large model pre-training system based on PYTORCH in this application. Figure 12 As shown, the system includes nodes 1-N, which are all source computing nodes. Text data is equally divided into data blocks 1-N and distributedly stored in nodes 1-N.

[0149] See also Figure 13 , Figure 13 This is a schematic diagram of a specific example of the text deduplication method provided by this application. Figure 13 As shown, text deduplication methods include:

[0150] 1. Each node performs a hash encoding on the text it stores to obtain the text encoding value of each text.

[0151] 1) For a text stored in a node, the node divides it into 128 text segments according to the NGram method.

[0152] 2) Perform hash coding on each of the 128 text segments to obtain the encoding sub-values ​​0-128 of the 128 text segments.

[0153] 3) Combine the encoding sub-values ​​0-128 of the 128 text segments, the first address information Node1ID of the text, and the second address information InnerID of the text to obtain the text encoding value. Node1ID identifies the node where the text is located. InnerID identifies the position of the text in the node. Figure 14 This is a diagram of the text encoding value of the text of this application.

[0154] 2. Each node divides the text encoding values ​​of the stored text into N types of encoding sets. The number of encoding set categories = the total number of nodes N.

[0155] Each node performs a secondary hashing on the text code values ​​of the stored text to obtain a category code value for each text. The node then divides the text code values ​​of each text based on the category code value. Text code values ​​corresponding to texts whose category code values ​​differ by less than a preset difference are grouped into the same code set.

[0156] Figure 15 This is a schematic diagram of the coding set obtained by dividing the four nodes in this application. Figure 15 As shown in the figure, there are nodes 1-4 (N=4). Node 1 divides the text encoding values ​​into encoding sets 11, 12, 13, and 14 for categories 1-4 based on the similarity between the text encoding values ​​of the text it stores. Similarly, node 2 divides the text encoding values ​​into encoding sets 21, 22, 23, and 24 for categories 1-4. Node 3 divides the text encoding values ​​into encoding sets 31, 32, 33, and 34 for categories 1-4. Node 4 divides the text encoding values ​​into encoding sets 41, 42, 43, and 44 for categories 1-4.

[0157] 3. Each node aggregates the same type of code sets to the same node and different types of code sets to different nodes according to the SCATTER strategy of PYTORCH's ring communication scheme.

[0158] Before aggregating various code sets, each node needs to convert the various code sets it divides into TENSOR data structures supported by PYTORCH for communication and transmission between different nodes.

[0159] Under PYTORCH's ring communication scheme SCATTER strategy, each node only communicates directly with its predecessor and successor nodes.

[0160] Continue reading Figure 15Node 1 retains the obtained code set 11 for category 1 and receives the code sets 21, 31, and 41 for category 1 sent by nodes 2-4. Node 2 retains the obtained code set 22 for category 2 and receives the code sets 12, 32, and 42 for category 2 sent by nodes 1, 3, and 4. Node 3 retains the obtained code set 33 for category 3 and receives the code sets 13, 23, and 43 for category 3 sent by nodes 1-2 and 4. Node 4 retains the obtained code set 44 for category 4 and receives the code sets 14, 24, and 34 for category 4 sent by nodes 1-3.

[0161] Thus, the coding set of category 1 obtained by dividing nodes 1-4 is aggregated to node 1, the coding set of category 2 obtained by dividing nodes 1-4 is aggregated to node 2, the coding set of category 3 obtained by dividing nodes 1-4 is aggregated to node 3, and the coding set of category 4 obtained by dividing nodes 1-4 is aggregated to node 4.

[0162] When the amount of text is large enough, it can be ensured that the number of text encoding values ​​received by each node is basically the same, so that the load remains balanced.

[0163] 4. Each node determines the repeated text group.

[0164] Node 1 combines the four encoding sets of category 1 to obtain a combination result of category 1; based on the Jaccard similarity between the text encoding values ​​in the combination result of category 1, the repeated text group corresponding to the combination result of category 1 is determined.

[0165] Node 2 combines the four code sets of category 2 to obtain a combination result of category 2; based on the Jaccard similarity between the text code values ​​in the combination result of category 2, the repeated text group corresponding to the combination result of category 2 is determined.

[0166] Node 3 combines the four code sets of category 3 to obtain a combination result of category 3; based on the Jaccard similarity between the text code values ​​in the combination result of category 3, the repeated text group corresponding to the combination result of category 3 is determined.

[0167] Node 4 combines the four code sets of category 4 to obtain a combination result of category 4; based on the Jaccard similarity between the text code values ​​in the combination result of category 4, the repeated text group corresponding to the combination result of category 4 is determined.

[0168] The calculation formula of the Gerard similarity between the text encoding values ​​of two texts in the combined result is as follows:

[0169] X={x1, x2, x3, ...x128};

[0170] Y={y1,y2,y3,...y128};

[0171] ;

[0172] Where X represents the text encoding value of one of the texts in the combination result, x1 represents the first encoding sub-value of X. Y represents the text encoding value of the other text, and y1 represents the first encoding sub-value of Y. represents the Jaccard similarity between X and Y, The number of encoded sub-values ​​representing the intersection of X and Y, The number of encoded subvalues ​​representing the union of X and Y.

[0173] Figure 16 It is a comparison diagram of the Jaccard similarity between the text encoding values ​​and the true similarity between texts. Figure 16 As shown in the figure, parti and partj represent the index of text i and text j respectively, linei and linej represent the line number of text i and text j in the source text data respectively, and value represents the specific content of text i and text j. real dis represents the real similarity between texts, and minhash represents the Jaccard similarity between the text encoding values ​​of texts. Figure 16 It can be seen that minhash can better represent the true similarity between texts.

[0174] When the Gerard similarity between the text encoding values ​​of two texts in the combination result is greater than the Gerard similarity threshold, the two texts are determined to be duplicate texts.

[0175] 5. Each node directly deletes the local text in the repeated text group, and notifies the node where the non-local text in the repeated text group is located to delete it.

[0176] The repeated text group determined by node 1 includes 8 repeated texts. Repeated texts 1-7 to be deduplicated are selected from the 8 repeated texts.

[0177] Node 1 determines the Node1ID and InnerID of each of the seven repeating texts based on the text encoding values ​​of repeating texts 1-7. The Node1IDs of repeating texts 1-7 represent Node 1, Node 1, Node 1, Node 2, Node 2, Node 3, and Node 4, respectively. That is, repeating texts 1-3 are stored in Node 1 (local text), repeating texts 4-5 are stored in Node 2 (non-local text), repeating text 6 is stored in Node 3 (non-local text), and repeating text 7 is stored in Node 4 (non-local text).

[0178] Node 1 directly deletes duplicate text 1-3.

[0179] After node 1 converts the InnerID of repeated texts 4-5, 6, and 7 into a format supported by the PYTORCH-based large model pre-training system, it sends it to nodes 2, 3, and 4 respectively, so that node 2 deletes repeated texts 4-5, node 3 deletes repeated text 6, and node 4 deletes repeated text 7.

[0180] It's understandable that related art text deduplication methods based on distributed communication systems determine the text to be deduplicated based on similarity between texts, requiring the transmission of text between nodes in the communication system. As the volume of text data increases, more nodes are needed to improve deduplication efficiency, but this also exponentially increases the amount of communication between nodes, leading to a decrease in both communication efficiency and stability.

[0181] In the above specific example of the present application, all texts are evenly divided into all nodes of PYTORCH. For each node, the node performs a hash code on each text therein into a text code value; the node performs a secondary hash code on each text code value into a category code value; the node divides the text code values ​​with the same category code value into the same category code set. The same category code set is aggregated into the same node, and the different category code sets are aggregated into different nodes to determine the text that needs to be deduplicated. The text code value carries the Node1ID and InnerID of the corresponding text. After determining the text that needs to be deduplicated, each node sends the InnerID of the text that needs to be deduplicated to the Node1ID node.

[0182] On the one hand, instead of directly calculating the similarity between texts, the similarity between text encoding values ​​is calculated, which can improve the deduplication efficiency and save computing overhead.

[0183] Furthermore, in order to determine the text that needs to be deduplicated, what is transmitted between nodes is the text encoding value and InnerID, so the amount of data that needs to be transmitted is small, which can further improve the deduplication efficiency.

[0184] Furthermore, the same type of code sets are aggregated to the same node to determine the nodes that need to be deduplicated, which can further improve the deduplication efficiency.

[0185] Furthermore, the text deduplication method is implemented based on Pytorch, eliminating the need to build a separate target communication system. Furthermore, due to the inherent performance of Pytorch, communication efficiency, stability, and scalability can be improved, further enhancing deduplication efficiency.

[0186] Figure 17 This is a schematic diagram of the structure of an embodiment of the text deduplication device of this application. Figure 17 As shown, the text deduplication device 70 includes an encoding module 71 , a division module 72 , and a deduplication module 73 .

[0187] The encoding module 71 is used to encode each text in the current source computing node once to obtain a text encoding value of each text.

[0188] The division module 72 is used to divide the text code values ​​of each text into several class code sets, wherein the similarity between the text code values ​​in the same class code set is higher than the similarity between different text code values ​​in different class code sets.

[0189] The deduplication module 73 is used to deduplicate the text corresponding to the encoding set in the current source computing node for each encoding set, wherein the text to be deduplicated is determined based on the similarity between the text encoding value of the text to be deduplicated and other texts in the encoding set.

[0190] For other detailed descriptions of the text deduplication device 70 of this embodiment, please refer to the previous embodiments, which will not be repeated here.

[0191] Figure 18 This is a schematic diagram of another embodiment of the text deduplication device of this application. Figure 18 As shown, the text deduplication device 80 includes a receiving module 81 , a combining module 82 , an acquiring module 83 and a determining module 84 .

[0192] The receiving module 81 is used to receive at least one type of coding set sent by at least one source computing node. The coding set is obtained by the source computing node dividing the text coding values ​​of each text in the source computing node. The text coding value of each text is obtained by the source computing node encoding each text separately.

[0193] A combining module 82 is used to combine the same type of code sets sent by each source computing node;

[0194] The acquisition module 83 is used to acquire the similarity between the text code values ​​in the combined code sets for each type of combined code sets.

[0195] The determination module 84 is configured to determine the text to be deduplicated in the combined code set based on the similarity, and notify the source computing node where the text to be deduplicated is located to delete the text to be deduplicated.

[0196] For other detailed descriptions of the text deduplication device 80 of this embodiment, please refer to the previous embodiments, which will not be repeated here.

[0197] Figure 19 This is a schematic diagram of the structure of an embodiment of the electronic device of the present application. Figure 19As shown, electronic device 90 includes memory 91 and processor 92. Processor 92 is configured to execute program instructions stored in memory 91 to implement the steps of any of the above method embodiments. In a specific implementation scenario, electronic device 90 may include, but is not limited to, a microcomputer and a server. Furthermore, electronic device 90 may also include a carrier device such as a laptop computer and a tablet computer, which are not limited herein.

[0198] Specifically, the processor 92 is used to control itself and the memory 91 to implement the steps in any of the above-mentioned method embodiments. The processor 92 can also be referred to as a CPU (Central Processing Unit). The processor 92 may be an integrated circuit chip with signal processing capabilities. The processor 92 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. In addition, the processor 92 can be implemented by an integrated circuit chip.

[0199] See also Figure 20 , Figure 20 The computer-readable storage medium 100 stores program instructions 101, which, when executed by a processor, implement the steps of any of the above method embodiments.

[0200] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0201] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0202] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. In another image position, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0203] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in either hardware or software functional units. If the integrated units are implemented as software functional units and sold or used as standalone products, they may be stored on a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the portion that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. This computer software product, stored on a storage medium, includes instructions for causing a computer device (such as a personal computer, server, or network device) or processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage media include various media capable of storing program code, such as USB flash drives, removable hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

Claims

1. A text deduplication method, characterized in that: include: Encoding each text in a current source computing node to obtain a text encoding value of each text, wherein the communication system in which the current source computing node is located is a target communication system, and multiple source computing nodes in the target communication system each contain multiple texts, and the current source computing node is one of the multiple source computing nodes; Dividing the text code values ​​of the texts into a plurality of code sets, wherein the similarity between the text code values ​​in the same code set is higher than the similarity between different text code values ​​in different code sets; For each type of the encoding set, deduplication is performed on the text corresponding to the encoding set in the current source computing node, wherein the text to be deduplicated is determined based on the similarity between the text encoding value of the text to be deduplicated and other text in the encoding set, including: At least one type of the coding set is used as a first type of coding set. For each of the first type of coding sets, the first type of coding sets in each of the source computing nodes are combined, and the similarity between the text coding values ​​in the combined first type of coding set is calculated. The texts corresponding to the text coding values ​​in the combined first type of coding set whose similarities meet the similarity requirements are used as a duplicate text group. At least one text is selected from the duplicate text group as a first text to be deduplicated, the first text to be deduplicated located at the current source computing node is deleted, and the source computing node where the non-local text is located is notified to delete the non-local text, wherein the non-local text is the first text to be deduplicated that is not located at the current source computing node; and / or At least one type of the coding set is used as the second type of coding set. For each of the second type of coding sets, the second type of coding set of the current source computing node is sent to the similarity computing node in the target communication system, so that the similarity computing node combines the second type of coding sets of each of the source computing nodes, and based on the similarity between the text coding values ​​in the combined second type of coding set, determines the second text to be deduplicated corresponding to the combined second type of coding set, and notifies the source computing node where the second text to be deduplicated is located to delete the second text to be deduplicated.

2. The method according to claim 1, characterized in that The step of encoding each text in the current source computing node to obtain a text encoding value of each text includes: For each of the texts, dividing the text into a plurality of text segments; Encoding each text segment of the text once respectively to obtain a coding sub-value of each text segment of the text; The text encoding value of the text is obtained by using the encoding sub-values ​​of each text segment of the text.

3. The method according to claim 2, characterized in that The one-time encoding method is hash encoding; And / or, obtaining the text encoding value of the text by using the encoding sub-values ​​of each text segment of the text includes: The encoding sub-values ​​of each text segment of the text and the position information of the text are combined to obtain the text encoding value of the text, wherein the position information of the text includes the first address information of the current source computing node where the text is located and the second address information of the text at the current source computing node.

4. The method according to any one of claims 1 to 3, characterized in that The text encoding values ​​of the texts are divided into several types of encoding sets, including: For each of the texts, performing secondary encoding on the text code value of the text to obtain the category code value of the text; The text encoding values ​​corresponding to the texts whose difference between the category encoding values ​​is less than a preset difference are divided into the same encoding set.

5. The method according to claim 4, characterized in that The secondary encoding method is hash encoding; And / or, dividing the text code values ​​corresponding to the texts whose difference between the category code values ​​is less than a preset difference into the same code set includes: The text code values ​​corresponding to the texts with the same category code values ​​are divided into the same category of code sets.

6. The method according to claim 1, characterized in that Among the several types of code sets, one type of code set is used as the first type of code set, and the other types of code sets are used as the second type of code sets; And / or, the similarity calculation node is the source calculation node; And / or, notifying a source computing node where the non-local text is located to delete the non-local text, or notifying a source computing node where the second to-be-deduplicated text is located to delete the second to-be-deduplicated text, includes: Using the non-local text or the second to-be-deduplicated text as the target deduplicated text; Acquire, from the text encoding value of the target deduplicated text, first address information of a source computing node where the target deduplicated text is located and second address information of the target deduplicated text in the source computing node; According to the first address information, the second address information is sent to the source computing node where the target deduplicated text is located, so as to notify the source computing node to delete the text corresponding to the second address information.

7. The method according to claim 1, characterized in that The text is pre-trained text for a large model; and / or The text deduplication method is executed by several nodes in a large model pre-training system based on PYTORCH, wherein the several nodes include the current source computing node; the method further includes: When data needs to be sent to other nodes, the data to be sent is converted into a format supported by the PYTORCH-based large model pre-training system.

8. A text deduplication method, characterized in that: include: Receiving at least one type of code set sent by at least one source computing node, wherein the code set is obtained by dividing the text code values ​​of each text in the source computing node by the source computing node, and the text code values ​​of each text are obtained by encoding the text by the source computing node respectively; Combining the same type of code sets sent by the source computing nodes; For each type of the combined code set, obtaining the similarity between the text code values ​​in the combined code set; Based on the similarity, the text to be deduplicated in the combined encoding set is determined, and the source computing node where the text to be deduplicated is located is notified to delete the text to be deduplicated.

9. The method according to claim 8, characterized in that The notifying the source computing node where the to-be-deduplicated text is located to delete the to-be-deduplicated text includes: Acquire, from the text encoding value of the to-be-deduplicated text, first address information of a source computing node where the to-be-deduplicated text is located and second address information of the to-be-deduplicated text in the source computing node; According to the first address information, the second address information is sent to the source computing node where the to-be-deduplicated text is located, so as to notify the source computing node to delete the text corresponding to the second address information.

10. A text deduplication device, characterized in that: include: an encoding module, configured to encode each text in a current source computing node once to obtain a text encoding value of each text, wherein the communication system where the current source computing node is located is a target communication system, and multiple source computing nodes in the target communication system each contain multiple texts, and the current source computing node is one of the multiple source computing nodes; a division module, configured to divide the text code values ​​of the texts into a plurality of code sets, wherein the similarity between the text code values ​​within the same code set is higher than the similarity between different text code values ​​between different code sets; A deduplication module is configured to dedupe text corresponding to the encoding set in the current source computing node for each encoding set, wherein the text to be deduplicated is determined based on the similarity between the text encoding value of the text to be deduplicated and other text in the encoding set, including: At least one type of the coding set is used as a first type of coding set. For each of the first type of coding sets, the first type of coding sets in each of the source computing nodes are combined, and the similarity between the text coding values ​​in the combined first type of coding set is calculated. The texts corresponding to the text coding values ​​in the combined first type of coding set whose similarities meet the similarity requirements are used as a duplicate text group. At least one text is selected from the duplicate text group as a first text to be deduplicated, the first text to be deduplicated located at the current source computing node is deleted, and the source computing node where the non-local text is located is notified to delete the non-local text, wherein the non-local text is the first text to be deduplicated that is not located at the current source computing node; and / or At least one type of the coding set is used as the second type of coding set. For each of the second type of coding sets, the second type of coding set of the current source computing node is sent to the similarity computing node in the target communication system, so that the similarity computing node combines the second type of coding sets of each of the source computing nodes, and based on the similarity between the text coding values ​​in the combined second type of coding set, determines the second text to be deduplicated corresponding to the combined second type of coding set, and notifies the source computing node where the second text to be deduplicated is located to delete the second text to be deduplicated.

11. A text deduplication device, characterized in that: include: a receiving module, configured to receive at least one type of code set sent by at least one source computing node, wherein the code set is obtained by dividing the text code values ​​of each text in the source computing node by the source computing node, and the text code values ​​of each text are obtained by encoding the text by the source computing node respectively; A combining module, configured to combine the same type of code sets sent by the source computing nodes; An acquisition module, configured to acquire, for each type of combined code set, the similarity between the text code values ​​in the combined code set; The determination module is used to determine the text to be deduplicated in the combined encoding set based on the similarity, and to notify the source computing node where the text to be deduplicated is located to delete the text to be deduplicated.

12. A target communication system, characterized in that: The method comprises several nodes, wherein the several nodes include at least one source computing node and at least one similarity computing node, the source computing node is used to execute the method according to any one of claims 1 to 7, and the similarity computing node is used to execute the method according to any one of claims 8 to 9.

13. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 9.

14. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images