A text similarity calculation method and system

Abstract

The application provides a text similarity calculation method and system. The method comprises the following steps: performing word segmentation on a first text to obtain a first word segmentation result, wherein the first word segmentation result comprises at least one first word segmentation and a word weight of each first word segmentation; for each first word segmentation, generating a weighted digital string of the first word segmentation according to a hash value and a word weight of the first word segmentation; and for each weighted digital string of the first word segmentation, performing vectorization accumulation calculation to obtain a sequence string of the first text, and performing vectorization dimension reduction on the sequence string of the first text to obtain a Simhash signature of the first text; obtaining a Simhash signature of a second text, and obtaining a similarity between the first text and the second text based on the Simhash signature of the first text and the Simhash signature of the second text. In this way, the calculation speed and overall efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method and system for calculating text similarity. Background Technology

[0002] With the development of computer and internet technologies, we often face massive amounts of data and information, posing significant challenges to data storage, processing, and transmission. This data and information often contains much identical or similar content. Deduplication and plagiarism detection can compress the data size for easier storage and processing. Furthermore, identifying completely identical or largely similar content facilitates classification and modeling, and provides reference data such as statistical data and plagiarism determination. Additionally, the party providing the original data and the party collecting the data for text similarity calculation also face challenges related to data sharing and privacy protection.

[0003] In the existing technology, there is a lack of an effective technical means to efficiently determine the similarity between two objects, such as two text contents, and provide a reliable quantitative indicator, while also taking into account aspects such as data sharing and privacy protection.

[0004] In summary, the current challenge is to provide a technical solution that balances data sharing and privacy protection while efficiently determining the similarity between two objects, such as two pieces of text, and providing reliable quantitative indicators. Summary of the Invention

[0005] This application provides a text similarity calculation method and system to solve the problems existing in the prior art.

[0006] Firstly, this application provides a method for calculating text similarity. The method includes: segmenting a first text into words to obtain a first segmentation result, the first segmentation result including at least one first segment and a word weight for each first segment; for each first segment, generating a weighted numeric string of the first segment according to its hash value and word weight; performing vectorized cumulative calculation on the weighted numeric string of each first segment to obtain a sequence string of the first text; and performing vectorized dimensionality reduction on the sequence string of the first text to obtain a Simhash signature of the first text; obtaining a Simhash signature of a second text, and based on the Simhash signature of the first text and the Simhash signature of the second text, obtaining the similarity between the first text and the second text.

[0007] Secondly, this application provides a system. The system includes: a word segmentation unit for segmenting a first text to obtain a first word segmentation result, the first word segmentation result including at least one first word and a word weight for each first word; a hash calculation unit for calculating the hash value of each word; a weighted numeric string generation unit for generating a weighted numeric string for each first word according to its hash value and word weight; an accumulation calculation unit for performing vectorized accumulation calculation on the weighted numeric strings of each first word to obtain a sequence string of the first text; and a dimensionality reduction unit for performing vectorized dimensionality reduction on the sequence string of the first text to obtain a Simhash signature of the first text.

[0008] Thirdly, embodiments of this application also provide a computing device, the computing device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a method according to any of the above-mentioned implementations.

[0009] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer instructions that, when executed on a computing device, cause the computing device to perform a method according to any of the above-described aspects.

[0010] Fifthly, embodiments of this application also provide a computer program product, characterized in that the computer program product includes instructions stored on a computer-readable storage medium, which, when executed on a computing device, cause the computing device to perform a method according to any of the above-described aspects. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram illustrating an example of the text similarity calculation process based on Simhash provided in this application embodiment;

[0013] Figure 2 A schematic diagram illustrating an application scenario for text similarity calculation based on Simhash, provided in an embodiment of this application;

[0014] Figure 3A schematic diagram illustrating another application scenario of Simhash-based text similarity calculation provided in this application embodiment;

[0015] Figure 4 A schematic diagram illustrating another application scenario of Simhash-based text similarity calculation provided in this application embodiment;

[0016] Figure 5 A flowchart illustrating a text similarity calculation method based on Simhash provided in this application embodiment;

[0017] Figure 6 Provided for the embodiments of this application Figure 5 A flowchart illustrating the vectorization-accelerated cumulative calculation and vectorization-accelerated dimensionality reduction steps of the text similarity calculation method;

[0018] Figure 7 Provided for the embodiments of this application Figure 6 A schematic diagram illustrating batch accumulation calculations using vector addition instructions in the vectorized accelerated accumulation calculation step;

[0019] Figure 8 A schematic diagram of a Simhash calculation module provided in an embodiment of this application;

[0020] Figure 9 Provided for the embodiments of this application Figure 8 A schematic diagram of the accumulation calculation unit of the Simhash calculation module;

[0021] Figure 10 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0023] It should be understood that in the description of this application, "at least one" means one or more, and "multiple" means two or more. In addition, the words "first," "second," etc., unless otherwise stated, are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0024] It should be understood that, in the description of this application, a binary digit (BIT) refers to the smallest unit of information, also known as a bit in binary digits. One bit of binary digits contains one unit of information. In computers, a bit is the smallest unit of data storage. Eight bits make up one byte, and one byte stores eight bits, which is also an 8-bit data unit. The main unit of measurement for information storage in computers is the byte; for example, one byte is 8 bits, one kilobyte (KB) is 1024 bits, and one megabyte (MB) is 1024 KB.

[0025] See Figure 1 , Figure 1 This diagram illustrates an example of the text similarity calculation process based on Simhash provided in this application. Simhash is a type of Locality Sensitive Hashing (LSH) algorithm that maps high-dimensional feature vectors to low-dimensional feature vectors, thereby determining whether texts are repeated or highly similar through the Hamming distance between the two vectors. Using the Simhash algorithm, representative keywords from multiple articles can be selected through word segmentation strategies, and these keywords are then encoded into fixed-length strings. By comparing these fixed-length encodings, multiple articles can be compared to determine whether they are identical or highly similar. The following example illustrates this process. Figure 1 The example shown illustrates the text similarity calculation process based on Simhash. Furthermore, the Hamming distance between two strings of equal length refers to the number of different characters at corresponding positions in the two strings, or the number of characters that need to be replaced to transform one string into the other. For two numbers, the Hamming distance between them is the number of different characters (or different binary values) at corresponding positions in the binary representation of the two numbers. For example, if two decimal numbers are 93 and 73, converting them to binary yields 101100 and 111000 respectively. It can be seen that the number of different characters at corresponding positions in the binary representations of the two strings is 2, therefore the Hamming distance between 93 and 73 is 2.

[0026] Step S102: Obtain the text "An employee in N district of country M said he saw a transport plane".

[0027] The text obtained in step S102 is the text content used for text similarity calculation. It can be data or information expressed in any form from any suitable source, such as a part of the content displayed on a webpage, feedback in a forum, or a paragraph in a paper. The text obtained in step S102 represents a part of a massive amount of information or data that needs to be checked for duplicates, deduplication, or similarity judgment. If text similarity calculation is performed on these texts by comparing them word by word, the calculation efficiency is low and the correctness of the judgment result is easily affected by typos, synonyms, etc. Traditional hash algorithms map the original content as evenly and randomly as possible to a signature value and compare the signature values ​​mapped from the text. If the signature values ​​of the two texts being compared are equal, it means that their original content is the same with a certain probability; otherwise, they are different. However, traditional hash algorithms can only indicate that the original content is different, but cannot measure the degree of similarity of the original content in the dimension of the signature. Even if the original content differs by one byte, it may lead to unequal signature values. The Simhash algorithm can provide a reference for characterizing the degree of similarity of the original content in the dimension of the signature. For example, consider two text strings: text string A reads "Your mom is calling you home for dinner, come home, come home," and text string B reads "Your mom is calling you home for dinner, come home, come home." It can be seen that these two text strings differ by only one character. The results obtained by calculating these two text strings using the traditional hash algorithm and the Simhash algorithm are shown in Table 1 below. Comparing the calculation results of text strings A and B under the traditional hash algorithm, it can be seen that the signature values ​​of the two text strings, which differ by only one character, have a very significant difference; that is, there are different binary digit values ​​at many corresponding positions. Conversely, comparing the calculation results of text strings A and B under the Simhash algorithm, it can be seen that there are only a few different binary digit values ​​at a few corresponding positions. Therefore, the calculation result under the Simhash algorithm can provide a reference for the similarity between text strings A and B in the signature dimension. In other words, the Hamming distance calculated by the Simhash algorithm provides a reference for the similarity between text strings A and B, and can be used to determine whether they are duplicates or highly similar.

[0028] Table 1

[0029]

[0030] Step S104: Segment the text “Employees in N district of M country said they saw a transport plane” to obtain the segments “M country” and “N district”, and mark the segment weights of “M country” and “N district” as 4 and 5 respectively.

[0031] In step S104, the word segmentation strategy is used to select representative keywords in the text. Therefore, the word segmentation strategy can be adjusted according to the actual application and requirements. In this example, based on the word segmentation strategy, the text "The clerk in area N of country M said that they saw a transport plane" is segmented into the word segment "country M" and the word segment "area N". In other examples, segmenting the text "The clerk in area N of country M said that they saw a transport plane" can result in the word segments "country M", "area N", "clerk", and "transport plane". In some embodiments, different texts from the same source can be based on the same word segmentation strategy or different word segmentation strategies. For example, for different paragraphs of the same paper, one word segmentation strategy can be applied to the previous paragraph and another word segmentation strategy can be applied to the subsequent paragraph. In some embodiments, the word segmentation strategy can be based on a machine learning (ML) model or a natural language processing (NLP) model or any suitable artificial intelligence model or algorithm. After segmenting the text, the word weight of each word segment is also identified, that is, the word weight identification is set. For example, assume that the word weight is divided into 5 levels, from the lowest level to the highest level are level 1, level 2, level 3, level 4, and level 5. Here, the identified word weight of the word segment "country M" is 4, that is, corresponding to level 4, and the word weight of the word segment "area N" is 5, that is, corresponding to level 5.

[0032] Step S106: Calculate the hash value of the word segment "country M" as "100101" and the hash value of the word segment "area N" as "101011" through the hash algorithm.

[0033] In step S106, calculate the hash values of each word segment obtained by word segmentation through the word segmentation strategy (such as through the NLP model) in step S104. Here, the hash algorithm can be common hash algorithms such as murmurhash, xxhash, cityhash, etc.

[0034] Step S108: Generate a weighted digital string according to the hash value and word weight of each word segment. The weighted digital string of the word segment "country M" is "4 -4-4 4 -4 4", and the weighted digital string of the word segment "area N" is "5 -5 5 -5 5 5".

[0035] In step S108, according to the word weights of each word segment obtained by word segmentation using a word segmentation strategy (such as through an NLP model) in step S104, and according to the hash values of each word segment obtained in step S106, the corresponding hash values and word weights are weighted and calculated to obtain a weighted digital string. Among them, the hash value of the word segment "Country M" is "100101" and its word weight is 4, and the weighted digital string of the word segment "Country M" obtained by weighted calculation is "4 -4-4 4 -4 4".

[0036] Step S110: Accumulatively calculate the weighted digital strings of each word segment of the text to obtain a sequence string of the text. Accumulatively calculate the weighted digital string "4 -4-4 4 -4 4" of the word segment "Country M" and the weighted digital string "5 -5 5 -5 5 5" of the word segment "Region N" to obtain "9 -9 1 -1 1 9".

[0037] In step S110, the weighted digital strings of each word segment of the text are accumulatively calculated to obtain a sequence string, which is the sequence string of the text. Among them, accumulative calculation means adding the numbers at the corresponding positions on the weighted digital string of each word segment. Therefore, the weighted digital string "4 -4-4 4 -4 4" of the word segment "Country M" and the weighted digital string "5 -55 -5 5 5" of the word segment "Region N" are accumulatively calculated to obtain "9 -9 1 -1 1 9".

[0038] Step S112: Reduce the dimension of the sequence string "9 -9 1 -1 1 9" of the text to obtain the Simhash signature "101011" of the text.

[0039] In step S112, the sequence string "9 -9 1 -1 1 9" of the text obtained by accumulative calculation in step S110 is changed into a binary digital string, that is, it is converted into a string expressed by 0 and 1, so as to achieve dimensionality reduction and obtain the Simhash signature. Specifically, each position of the sequence string of the text is compared. If the number at a certain position is greater than 0, it is recorded as 1. If the number at a certain position is less than 0, it is recorded as 0. Therefore, the sequence string "9 -9 1 -1 1 9" of the text undergoes dimensionality reduction processing to obtain the Simhash signature "101011" of the text.

[0040] It should be understood that the above steps S102 to S112 take the given text "The clerk in Country M and Region N said that they saw a transport plane" as an example and illustrate how to gradually calculate the Simhash signature of the given text. The calculation process of text similarity calculation based on Simhash and the vectorization method of text similarity calculation based on Simhash mentioned in the embodiments of this application can be applied to any input text and any hash algorithm can be used to calculate the hash value of word segments.

[0041] Referring to steps S102 to S112 above, an example of the text similarity calculation process based on Simhash is provided. The input text is segmented and weighted, and then the hash value of each segment is calculated, along with its weighted sum, to obtain the hash weighted result for each segment. Next, the hash weighted results of all segments are summed bit-by-bit to obtain the accumulated result. Finally, the accumulated result is processed bit-by-bit, taking values ​​of 0 or 1, thus obtaining the Simhash signature of the text. In practice, depending on the segmentation strategy used and the input text, different data streams can be generated. In particular, the step of summing the hash weighted results of all segments may face complex and varied computational requirements. For example, in step S110, the weighted numerical strings of each word segment of the text are accumulated to obtain the text sequence string. The weighted numerical strings of words with higher weight levels may have larger numbers, while those of words with lower weight levels may have smaller numbers. Furthermore, the length of the weighted numerical strings is difficult to predict. Such computational requirements suffer from slow computation speed when performed using traditional scalar addition instructions or operations. Therefore, it is necessary to provide an optimized acceleration method to improve computation speed and running efficiency, considering the characteristics of the data flow generated in the bitwise accumulation step of the hash weighted results of all words in the Simhash-based text similarity calculation process, as well as the algorithmic characteristics of the Simhash-based text similarity calculation process. This will be further explained below with reference to specific embodiments.

[0042] See Figure 2 , Figure 2 This diagram illustrates an application scenario for text similarity calculation based on Simhash, as provided in an embodiment of this application. (Reference) Figure 1 The example shown illustrates the computational process for text similarity calculation based on Simhash. The Simhash algorithm provides a reference for the degree of similarity between the original content at the signature level. It calculates the Simhash signature for each of the two objects to be compared. For example, it calculates the Simhash signatures for text strings A and B as shown in Table 1. Then, by comparing the Hamming distance between their Simhash signatures (i.e., the number of different binary digit values ​​at corresponding positions in their respective Simhash signatures), a reference for the degree of similarity between the two objects can be provided at the signature level. This can be used to determine whether two objects are duplicates or highly similar. Figure 2 In the application scenario shown, the two or more objects used for comparison come from a webpage, for example... Figure 2Web pages 202, 204, and 206 are shown. These web pages are collected and organized via the Internet 210. Then, through operation 220, which involves using search engines and web crawlers to obtain the content of each web page, operation 230 is performed to calculate the similarity of the web page content and obtain a Simhash signature. In operation 220, search engines and web crawlers or similar software can frequently obtain various web pages from the Internet 210. The content of these web pages is often duplicated or highly similar. Calculating the Simhash signatures of the web page content in operation 230 is beneficial for subsequent deduplication, deduplication checking, modeling, and statistical analysis. For example, identifying a large amount of similar content in web pages can provide reference for trend judgment, public opinion analysis, or business decision-making. The web page content and the Simhash signatures calculated based on the web page content are stored in a web page database 240. The data stored in the web page database 240 can be used for operation 250, which is a web page deduplication service, thus avoiding the storage of identical or highly similar web page content. Besides being used for webpage deduplication services, it can also be used for other services such as modeling, statistics, and analysis. The Simhash signature of webpage content provides a reference for the similarity between the content of different webpages at the signature level, which can be used to quickly and effectively determine whether the content of different webpages is duplicated or highly similar. Furthermore, because the webpage deduplication service, i.e., operation 250, is based on the Simhash signature of the webpage content, the server or host performing the webpage deduplication service does not need to access the original data of the webpage content. Therefore, it is beneficial to protect the data security and privacy of the party providing the original data of the webpage content (e.g., webpage library 240).

[0043] See Figure 3 , Figure 3 This diagram illustrates another application scenario for Simhash-based text similarity calculation provided in this application. (Reference) Figure 1 The example shown illustrates the computational process for text similarity calculation based on Simhash. The Simhash algorithm provides a reference for the degree of similarity between the original content at the signature level. It calculates the Simhash signature for each of the two objects to be compared. For example, it calculates the Simhash signatures for text strings A and B as shown in Table 1. Then, by comparing the Hamming distance between their Simhash signatures (i.e., the number of different binary digit values ​​at corresponding positions in their respective Simhash signatures), a reference for the degree of similarity between the two objects can be provided at the signature level. This can be used to determine whether two objects are duplicates or highly similar. Figure 3 In the application scenario shown, the two or more objects used for comparison come from a webpage, for example... Figure 3Papers 302, 304, and 306, as shown, are compiled and organized through journals, conferences, and reports (310). Then, operation 320, which involves academic retrieval and journal retrieval, retrieves the content of each paper. Next, operation 330 calculates the similarity of the paper content to obtain its Simhash signature. In operation 320, academic retrieval, journal retrieval, or similar retrieval services retrieve various papers from journals, conferences, and reports (310). The content of these papers needs to undergo deduplication and plagiarism checks to ensure academic quality and identify plagiarism. Calculating the Simhash signatures of these papers in operation 330 facilitates subsequent deduplication, plagiarism detection, modeling, and statistical analysis. For example, when the Hamming distance between the Simhash signatures of two papers is very short, it may indicate that the two papers are highly similar and can be marked for subsequent manual processing. This allows for rapid deduplication and plagiarism checks on multiple papers. The paper content and the Simhash signature calculated based on it are stored in paper repository 340. The data stored in repository 340 can be used for operation 350, i.e., the paper plagiarism detection service. This allows for fast and efficient plagiarism detection and identification of academic plagiarism among multiple papers from journals, conferences, and reports 310 stored in repository 340. Besides plagiarism detection, it can also be used for other services such as modeling, statistics, and analysis. The Simhash signature of the paper content provides a reference for the similarity between the content of different papers at the signature level, which can be used to quickly and effectively determine whether the content of different papers is duplicated or highly similar. Furthermore, because the plagiarism detection service, i.e., operation 350, is based on the Simhash signature of the paper content, the server or host performing the plagiarism detection service does not need to access the original data of the paper content. This is beneficial for protecting the data security and privacy of the party providing the original data (e.g., repository 340).

[0044] See Figure 4 , Figure 4 This diagram illustrates another application scenario for Simhash-based text similarity calculation provided in this application. (Reference) Figure 1The example shown illustrates the computational process for text similarity calculation based on Simhash. The Simhash algorithm provides a reference for the degree of similarity between the original content at the signature level. It calculates the Simhash signature for each of the two objects to be compared. For example, it calculates the Simhash signatures for text strings A and B as shown in Table 1. Then, by comparing the Hamming distance between their Simhash signatures (i.e., the number of different binary digit values ​​at corresponding positions in their respective Simhash signatures), a reference for the degree of similarity between the two objects can be provided at the signature level. This can be used to determine whether two objects are duplicates or highly similar. Figure 4 In the application scenario shown, the two or more objects used for comparison come from user feedback, for example... Figure 4The user feedback examples 402, 404, and 406 shown are collected and organized through product and e-commerce user communities and forums 410. Then, operation 420 retrieves the content of each user feedback through forum and post retrieval. Next, operation 430 calculates the similarity of the user feedback content to obtain its Simhash signature. In operation 420, forum and post retrieval services, or similar retrieval services, retrieve user feedback from various product and e-commerce user communities and forums. Identifying similar content facilitates categorization of this feedback, which can then be used for subsequent purposes such as public opinion management and analysis. Calculating the Simhash signature of the user feedback content in operation 430 facilitates subsequent deduplication, deduplication, modeling, and statistical analysis. For example, by counting the number or scale of user feedback with sufficiently low similarity, representative user feedback can be identified and analyzed to generate reference information such as public opinion, sentiment analysis, and public opinion guidance. User feedback and the Simhash signatures calculated based on the user feedback content are stored in the user feedback database 440. The data stored in the user feedback database 440 can be used for operation 450, which is the user feedback deduplication service. This allows for the rapid and efficient deduplication and classification of user feedback from various product, e-commerce, user communities, forums, and other sources stored in the user feedback database 440, thus facilitating subsequent data analysis and processing. The Simhash signature of the user feedback content provides a reference for the similarity between different user feedback contents at the signature level, which can be used to quickly and effectively determine whether the user feedback content of different user feedbacks is duplicated or highly similar. Furthermore, because the user feedback deduplication service, i.e., operation 450, is based on the Simhash signature of the user feedback content, the server or host performing the user feedback deduplication service does not need to access the original data of the user feedback content. Therefore, it is beneficial to protect the data security and privacy of the party providing the original data of the user feedback content (such as the user feedback database 440).

[0045] In combination with the above Figures 1 to 4 The example of the text similarity calculation process based on Simhash utilizes the Simhash algorithm to provide a reference representing the degree of similarity of the original content at the signature dimension, such as... Figure 2 The application scenarios of webpage deduplication shown are as follows: Figure 3 The application scenarios of paper plagiarism detection shown are as follows: Figure 4 The user feedback deduplication application scenarios shown provide a reliable quantitative indicator for determining the similarity between two objects. It can be seen that the specific objects addressed in the Simhash-based text similarity calculation process are determined by the application scenario; for example, in... Figure 2The application scenario of deduplication shown requires determining the degree of similarity between the content of different web pages. Figure 3 The application scenario of plagiarism detection shown requires determining the degree of similarity between the content of different papers. Figure 4 The user feedback deduplication application scenario shown requires determining the similarity between different user feedback contents. Therefore, depending on the specific application scenario, the Simhash-based text similarity calculation process may face different input texts or different objects used for text similarity calculation, and may employ different word segmentation strategies or NLP models. This makes the data flow in the calculation process complex and variable, making it difficult to optimize using scalar addition instructions. Therefore, it is necessary to provide optimized acceleration methods to improve computation speed and running efficiency, considering the characteristics of the data flow generated in the bitwise accumulation step of the hash weighted results of all word segments in the Simhash-based text similarity calculation process, as well as the algorithmic characteristics of the Simhash-based text similarity calculation process. The following section combines... Figure 5 The embodiments are further illustrated.

[0046] See Figure 5 , Figure 5 This is a flowchart illustrating a text similarity calculation method based on Simhash, provided as an embodiment of this application. Figure 5As shown, the vectorization method includes the following steps. It should be understood that the text similarity calculation method and system provided in this application embodiment can be used to calculate the similarity of multiple texts, such as calculating the similarity of a first text and a second text. The terms "first text," "second text," etc., mentioned in this application embodiment are for convenience only and do not necessarily involve two or more texts. The text similarity calculation method and system provided in this application embodiment can also involve only a single text and generate a Simhash signature for that single text. This generated Simhash signature can be used to compare the similarity with the Simhash signatures of other texts, and the Simhash signatures of other texts can be generated in any other suitable way. In other words, the text similarity calculation method and system provided in this application embodiment provides a technical solution that balances data sharing and privacy protection, and can efficiently determine the degree of similarity between two objects, such as the content of two texts, and provide reliable quantitative indicators. The quantitative indicator generated through this application embodiment, such as the first Simhash signature of the first text, can be used to calculate the similarity with the second Simhash signature of the second text, which is also generated through this application embodiment, thereby determining the degree of similarity between the contents of the first text and the second text. However, the quantitative indicators generated through the embodiments of this application, such as the first Simhash signature of the first text, can also be used to calculate similarity with Simhash signatures generated by other methods. The terms "first text," "second text," etc., mentioned below are for ease of description and to distinguish different texts only, unless the context otherwise requires.

[0047] Step S502: Obtain the first text for text similarity calculation.

[0048] In step S502, the first text used for text similarity calculation can be a webpage, a paper, user feedback, or any suitable medium or form of expression. Text similarity calculation provides a reference for determining whether texts are repetitive or highly similar and provides quantitative indicators to indicate the degree of similarity between texts. For example, assuming the text obtained in step S502 for text similarity calculation is webpage content from different webpages, then text similarity calculation is used to indicate whether the content of different webpages is repetitive or highly similar, such as providing a webpage deduplication service.

[0049] Step S504: Perform word segmentation on the first text to obtain a first word segmentation result, wherein the first word segmentation result includes at least one first word and the word weight of each first word.

[0050] In step S504, the first text can be tokenized according to a specific tokenization strategy, NLP model or algorithm. For example, representative keywords can be screened out from the first text and parts that do not contain useful information can be ignored. Moreover, the specific way of tokenizing the first text, such as the tokenization strategy or NLP model adopted, can be specific to the first text. In some embodiments, different input texts from the same source, such as different paragraphs from the same paper, can adopt different tokenization strategies respectively. For example, one tokenization strategy can be adopted for the previous paragraph or the previous text, and another tokenization strategy can be adopted for the subsequent paragraph or the subsequent text. In addition, for each of the first tokens, the word weights are respectively identified, and the specific way of word weight identification can also be specific to the first text. For example, assuming that the word weights are divided into 5 levels, from the lowest level to the highest level are level 1, level 2, level 3, level 4 and level 5 respectively, and the word weights from the lowest level to the highest level are correspondingly set as 1, 2, 3, 4 and 5.

[0051] Step S506: Traverse and calculate the hash value of each of the at least one token through a hash algorithm.

[0052] In step S506, the hash algorithm can be a common hash algorithm such as murmurhash, xxhash, cityhash, etc.

[0053] Step S508: For each of the first tokens, generate a weighted digital string of the first token according to the hash value and word weight of the first token.

[0054] In step S508, the hash value and word weight of each of the first tokens are respectively weighted calculated to obtain a weighted digital string. For example, assuming that the hash value of the token "Country M" is "100101" and its word weight is 4, the weighted digital string of the token "Country M" obtained through weighted calculation is "4 -4-4 4 -4 4".

[0055] Step S510: For the weighted digital string of each of the first tokens, perform vectorized accumulation calculation to obtain a sequence string of the first text, and perform vectorized dimensionality reduction on the sequence string of the first text to obtain the Simhash signature of the first text.

[0056] In step S510, the weighted numeric strings of each segmented word are summed, meaning the numbers at corresponding positions in the weighted numeric strings of each segmented word are summed. The result of the summation is then converted into a binary digitized string, i.e., dimensionality reduction is performed. As mentioned above, depending on the specific application scenario, the text similarity calculation process based on Simhash may face different input texts or different objects used for text similarity calculation, and may employ different word segmentation strategies or NLP models. This makes the data flow in the calculation process complex and variable. For example, the weighted numeric strings that need to be summed in step S510 may have varying lengths, and the numbers in the weighted numeric strings may also vary within a large range (affected by word weight identifiers and word weight levels). Scalar operations can only perform scalar operations on a pair of data at a time. To complete the summation of two weighted numeric strings, multiple scalar operations are required to sum the numbers at corresponding positions in both weighted numeric strings. Vectorization accelerates accumulation computation. For example, using instructions that support vector operations, such as the Single Instruction Multiple Data (SIMD) instruction set, the same operation can be performed on each data point in a set of data (or data vector), thus achieving spatial parallelism. For instance, the accumulation computation of two weighted number strings can be accelerated by vectorization, such as using SIMD instructions to simultaneously process the accumulation of numbers at multiple corresponding positions on the two weighted number strings. In some embodiments, SIMD instructions can use a single controller to control multiple processors, thereby operating on multiple registers, with each register used to store a smaller integer in a compressed format.

[0057] After obtaining the sequence string of the first text through vectorized accelerated accumulation calculation, the sequence string of the first text is further vectorized for accelerated dimensionality reduction to obtain the Simhash signature of the first text. Vectorized accelerated dimensionality reduction can also be based on instructions supporting vector operations, such as the SIMD instruction set. Thus, by using instructions supporting vector operations, such as the SIMD instruction set, or similar control mechanisms, the scalar operations in the bitwise accumulation and dimensionality reduction stages of the Simhash-based text similarity calculation process are transformed into vector operations. This facilitates the execution of batch data addition and batch data comparison within a single cycle, thereby improving computational speed and operational efficiency. The following section combines... Figure 6 The embodiments are further described in detail. Figure 5 The vectorization method accelerates the cumulative calculation and the dimensionality reduction steps.

[0058] In addition, through Figure 5After obtaining the Simhash signature of the first text using the text similarity calculation method shown, step S512 can be performed: obtaining the Simhash signature of the second text, and based on the Simhash signatures of the first and second texts, obtaining the similarity between the first and second texts. In some embodiments, the method for obtaining the Simhash signature of the second text can refer to the steps or principles for obtaining the Simhash signature of the first text, and will not be repeated here.

[0059] See Figure 6 , Figure 6 Provided for the embodiments of this application Figure 5 The flowchart illustrates the vectorized acceleration of cumulative calculation and vectorized acceleration of dimensionality reduction steps in the text similarity calculation method. (See attached diagram.) Figure 6 As shown, Figure 5 The vectorized accelerated accumulation calculation and vectorized accelerated dimensionality reduction expansion in step S510 (for each weighted numeric string of the segmented words, perform vectorized accumulation calculation to obtain the sequence string of the first text, and perform vectorized accelerated dimensionality reduction on the sequence string of the first text to obtain the first Simhash signature of the first text) result in the following steps.

[0060] Step S610: Initialize the first register YMM0 according to the first integer value length, which is based on the first word segmentation result, which includes at least one word, and the weighted numeric string of each of the at least one word is converted into a byte array, the length of which is the first integer value length.

[0061] Instructions supporting vector operations, such as the SIMD instruction set, define multiple registers. SIMD instructions allow a single controller to manage multiple processors, enabling operations on multiple registers. For example, the Advanced Vector Extensions (AVX) instruction set for x86 architecture microprocessors includes 16 256-bit wide registers. Registers used for vector operations, such as vector registers, use various data structures as operand types. Before operating on multiple registers using SIMD instructions, the registers need to be initialized and the operand types specified. For example, the AVX instruction set includes registers providing various operand types, including 32-bit or 4-byte single-precision floating-point numbers, 64-bit or 8-byte double-precision floating-point numbers, and integer types (potentially supporting 8-bit or 1-byte, 16-bit or 2-byte, 32-bit or 4-byte, 64-bit or 8-byte). In step S610, the first register YMM0 is initialized according to the first integer type value length. This can be done by initializing the first register YMM0 used for vector operations using a specified integer type (e.g., setting the operand type to 1 byte, 2 bytes, 4 bytes, or 8 bytes). Furthermore, the first integer numerical length is based on the first word segmentation result, which includes at least one word. The weighted numeric string of each of these at least one word is converted into a byte array, the length of which is the first integer numerical length. As mentioned above, the length of the weighted numeric string requiring accumulation calculation may be complex and variable, and the numbers in the weighted numeric string may also vary within a large range (affected by word weight identifiers and word weight levels). Therefore, the computational requirements of the weighted numeric strings of the words included in the first word segmentation result can be reflected by setting a corresponding first integer numerical length. Additionally, the fact that the weighted numeric string of each of the words included in the first word segmentation result is converted into a byte array with the first integer numerical length means that all words in the first word segmentation result can be processed through the first register YMM0 initialized according to the first integer numerical length.

[0062] After executing step S610, proceed to step S620.

[0063] Step S620: Traverse the weighted numeric string of each of the at least one word segment, and load the byte array corresponding to a specific number of weighted numeric strings into the second register YMM1 each time. The specific number is determined based on the length of the first integer value and the width of the second register YMM1. The width of the second register YMM1 is equal to the width of the first register YMM0.

[0064] In step S620, the width of the second register YMM1 is equal to the width of the first register YMM0, therefore all segments of the first segmentation result are also applicable to the second register YMM1. Considering that all segments of the first segmentation result may not be loaded into the second register YMM1 at once, a specific number of byte arrays corresponding to weighted numeric strings are loaded into the second register YMM1 each time it is traversed. For example, one, two, three, or other numbers of byte arrays corresponding to weighted numeric strings are loaded into the second register YMM1 each time it is traversed. This specific number is determined based on the length of the first integer value and the width of the second register YMM1. For example, if the width of the second register YMM1 is 256 bits and the length of the first integer value is 32 bits, then 8 byte arrays corresponding to weighted numeric strings are loaded into the second register YMM1 each time it is traversed. For example, the SIMD instruction set supports loading 256-bit integer data into a vector register without memory alignment. As mentioned above, the weighted numeric string of each of the at least one segmented word is converted into a byte array. The length of this byte array is the length of the first integer value. The computational requirements of the weighted numeric strings of the segmented words included in the first segmentation result are reflected by setting the corresponding first integer value length. Thus, by determining this specific number based on the first integer value length and the width of the second register YMM1, the data loading operation on the second register YMM1 also reflects the computational requirements of the weighted numeric strings of the segmented words included in the first segmentation result, and can be flexibly adapted by combining the widths of the first register YMM0 and the second register YMM1.

[0065] After executing step S620, proceed to step S630.

[0066] Step S630: Perform batch accumulation calculation on the specific number of weighted number strings stored in the first register YMM0 and the second register YMM1 respectively using a vector addition instruction to obtain the accumulation calculation result, and store the accumulation calculation result in the first register YMM0, wherein the vector addition instruction is based on the length of the first integer value.

[0067] As mentioned above, in step S610, the first register YMM0 is initialized according to the first integer value length, and in step S620, a specific number of byte arrays corresponding to weighted number strings (determined based on the first integer value length and the width of the second register YMM1) are loaded into the second register YMM1. Furthermore, the width of the second register YMM1 is equal to the width of the first register YMM0. This means that in step S630, the first register YMM0 and the second register YMM1 can each store the same number (i.e., the specific number) of weighted number strings. Consequently, a vector addition instruction can be used to perform batch data addition on the weighted number strings stored in the first register YMM0 and the second register YMM1, thus performing bitwise accumulation using vector addition. In other words, multiple weighted number strings can be accumulated simultaneously in one cycle. It should be understood that the vector addition instruction is based on the first integer value length, which can be used as a parameter to control the vector addition instruction. For example, the SIMD instruction set supports adding the values ​​in two registers of the same width as integer values ​​of 1 byte, 2 bytes, 4 bytes, or 8 bytes and storing the result in the register. Furthermore, by setting the appropriate first integer value length to reflect the computational requirements of the weighted numeric string of the first segmented result, it can handle complex and varied data streams caused by different input texts, different segmentation strategies, or NLP models.

[0068] After executing step S630, proceed to step S640.

[0069] Step S640: Determine whether to traverse each of the at least one word. If the determination is yes in step S640, proceed to step S650; if the determination is no in step S640, proceed to step S620.

[0070] If step S640 determines that the condition is negative, meaning that each of the at least one word has not been traversed, then step S620 is executed to perform the next traversal and load the byte array corresponding to the new specific number of weighted numeric strings into the second register YMM1. Then, in step S630, the new accumulated calculation result is stored in the first register YMM0. Thus, when the traversal ends, meaning each of the at least one word has been traversed, the first register YMM0 stores the accumulated calculation result of all the words of the at least one word.

[0071] Step S650: Initialize the second register YMM1 according to the length of the first integer value.

[0072] As mentioned above, after the traversal is complete, the first register YMM0 stores the cumulative calculation result of all segments of the at least one segment. In step S650, the second register YMM1 is initialized according to the length of the first integer value. This can be done by initializing the second register YMM1 used for vector operations using a specified integer type (e.g., setting the operand type to 1 byte, 2 bytes, 4 bytes, or 8 bytes). Thus, the initialized second register YMM1 can be used for subsequent batch data comparisons with the first register YMM0.

[0073] After executing step S650, proceed to step S660.

[0074] Step S660: Perform a batch comparison between the first register YMM0 and the second register YMM1 using a batch comparison instruction to obtain a comparison result, and store the comparison result in the first register YMM0, wherein the batch comparison instruction is based on the length of the first integer value.

[0075] In step S660, the first register YMM0 stores the cumulative calculation result of all segments of the at least one word, while the second register YMM1 is initialized, i.e., cleared to zero, in step S650. (Refer to the above.) Figure 1 This example illustrates the computational process for text similarity calculation based on Simhash. After obtaining the text sequence, it needs to be converted into a binary digitized string, that is, a string represented by 0s and 1s, to achieve dimensionality reduction and obtain the Simhash signature. Specifically, the text sequence is compared position by position. If the number at a certain position is greater than 0, it is recorded as 1; if the number at a certain position is less than 0, it is recorded as 0. Therefore, in step S660, through batch comparison instructions, such as the SIMD instruction set based on support vector operations, the values ​​in two registers of the same width can be divided into multiple positions according to the length of the first integer value, and the values ​​of the two registers at the corresponding positions can be compared, with a 1 or 0 returned to indicate the comparison result. Thus, in step S660, batch comparison is performed through batch comparison instructions, and the comparison results are stored in the first register YMM0, realizing that the dimensionality reduction process is performed in a vector operation manner.

[0076] After executing step S660, proceed to step S670.

[0077] Step S670: Save the storage result of the first register YMM0 to a byte array of length equal to the length of the first integer value to generate the Simhash signature of the first word segmentation result.

[0078] In step S670, the storage result of the first register YMM0 is saved to a byte array of the same length as the first integer value. This can be achieved using, for example, an operation instruction provided by the SIMD instruction set for saving the value stored in the register to an integer array in memory, such as moving a 256-bit integer vector to a 256-bit memory-aligned position. It should be understood that after saving the storage result of the first register YMM0 to a byte array of, for example, a length of 32 bits (the length of the first integer value is 32 bits), a left shift and logical AND operation can be used to combine the 32-bit byte array into a 32-bit integer Simhash value, which is the Simhash signature of the first tokenization result.

[0079] See Figure 7 , Figure 7 Provided for the embodiments of this application Figure 6 This diagram illustrates the batch accumulation calculation performed using vector addition instructions in the vectorized accelerated accumulation calculation step. (See diagram for example.) Figure 7 As shown, the vector stored in the first register YMM0 contains four data points: X1, X2, X3, and X4. The vector stored in the second register YMM1 contains four data points: Y1, Y2, Y3, and Y4. The cumulative calculation results obtained by batch accumulation using vector addition instructions show that four data points can be processed simultaneously within one cycle, resulting in vectors of X1+Y1, X2+Y2, X3+Y3, and X4+Y4.

[0080] See Figure 6 and Figure 7 By employing vector operation instructions, such as those supporting vector operations in the AVX instruction set of x86 architecture microprocessors, the system achieves batch loading of hash weighted results (i.e., weighted numeric strings) to be accumulated, and completes batch data accumulation within a single clock cycle. This vectorization optimization addresses the characteristics of the data flow generated during the bitwise accumulation of hash weighted results across all words in the Simhash-based text similarity calculation process, as well as the algorithmic characteristics of the Simhash-based text similarity calculation process, significantly improving computation speed. Furthermore, for the complex and varied computational needs of applications requiring Simhash calculations on massive amounts of text such as web pages, papers, and user feedback, the system sets appropriate integer length values ​​to reflect the impact of specific word segmentation strategies and NLP models on the data flow, thus flexibly adapting to various application scenarios and improving overall performance. Additionally, batch comparison instructions achieve batch data comparison, also implementing vectorization optimization for the dimensionality reduction stage, and reusing register resources used for bitwise accumulation, improving resource utilization efficiency.

[0081] It should be understood that the aforementioned vector addition instructions and other instruction sets supporting vector operations are exemplified by the AVX instruction set of x86 architecture microprocessors. Other instruction sets, as long as they support the vectorization method provided in the embodiments of this application, can also be applied to this application, such as ARM architecture microprocessors (e.g., the NEON instruction set of ARM architecture microprocessors), PowerPC architecture microprocessors, SPARC architecture microprocessors, MIPS architecture microprocessors (e.g., the MSA instruction set of MIPS architecture microprocessors), etc.

[0082] It should be understood that the first register YMM0 and the second register YMM1 mentioned above are merely illustrative. In actual applications, the names of the registers used can be determined according to the specific instruction set used, and the compiler can specify the specific registers to be used. Therefore, as long as the operating principle mentioned in the embodiments of this application is satisfied, any register or register name specified by the instruction set or the compiler can be used to correspond to the first register YMM0 and / or the second register YMM1 mentioned above.

[0083] See above Figures 1 to 7 This application provides a text similarity calculation method. The method includes: segmenting a first text into words to obtain a first segmentation result, the first segmentation result including at least one first segment and a word weight for each first segment; for each first segment, generating a weighted numeric string of the first segment according to its hash value and word weight; performing vectorized cumulative calculation on the weighted numeric string of each first segment to obtain a sequence string of the first text, and performing vectorized dimensionality reduction on the sequence string of the first text to obtain a Simhash signature of the first text; obtaining a Simhash signature of a second text, and obtaining the similarity between the first text and the second text based on the Simhash signatures of the first and second texts. Thus, this method accelerates the cumulative calculation and dimensionality reduction through vectorization. It addresses the characteristics of the data flow generated in the step of accumulating the hash weighted results of all segments in the Simhash-based text similarity calculation process, as well as the algorithmic characteristics of the Simhash-based text similarity calculation process, achieving vectorization optimization and significantly improving the calculation speed.

[0084] It should be understood that, Figure 5 and Figure 6The method embodiment obtains a first text for text similarity calculation and generates a first Simhash signature of the first text by accelerating the accumulation calculation and dimensionality reduction using vectorization. The above method embodiment can also be applied to a second text and generate a second Simhash signature of the second text by accelerating the accumulation calculation and dimensionality reduction using vectorization. The above method embodiment can be applied to multiple texts, including a first text and a second text, generating Simhash signatures for each of the multiple texts by accelerating the accumulation calculation and dimensionality reduction using vectorization, thereby enabling similarity comparison. Therefore, it achieves a technical solution that balances data sharing and privacy protection, and can efficiently determine the similarity between two objects, such as two text contents, and provide reliable quantitative indicators.

[0085] In one possible implementation, the step of performing vectorized accelerated accumulation calculation to obtain the sequence string of the first text includes: traversing the weighted numeric string of each of the at least one word segment, loading the byte array corresponding to a specific number of weighted numeric strings into the second register each time, and performing batch accumulation calculation on the weighted numeric strings stored in the first register and the second register respectively through vector addition instructions to obtain the accumulation calculation result, and then storing the accumulation calculation result into the first register.

[0086] In one possible implementation, the specific number is determined based on the length of a first integer value and the width of the second register, the width of which is equal to the width of the first register.

[0087] In one possible implementation, the first integer numerical length is based on the first word segmentation result of the first text, and the length of the byte array corresponding to the weighted number string of each of the at least one word is the first integer numerical length.

[0088] In one possible implementation, the first register is initialized with the length of the first integer value at least before the vectorized accelerated accumulation calculation is performed.

[0089] In one possible implementation, the vector addition instruction is based on the length of the first integer value.

[0090] In one possible implementation, the second register is initialized according to the length of the first integer value after the vectorized accelerated accumulation calculation and before the vectorized accelerated dimensionality reduction of the sequence string of the first text.

[0091] In one possible implementation, the vectorization of the sequence string of the first text to accelerate dimensionality reduction includes: performing a batch comparison between the first register and the second register using a batch comparison instruction to obtain a comparison result, and storing the comparison result in the first register.

[0092] In one possible implementation, the batch comparison instruction is based on the length of the first integer value.

[0093] In one possible implementation, the vectorization of the sequence string of the first text to accelerate dimensionality reduction further includes: saving the storage result of the first register into a byte array of length equal to the length of the first integer value to generate the Simhash signature of the first text.

[0094] In one possible implementation, the vectorization of the sequence string of the first text to accelerate dimensionality reduction further includes: using left shift and logical AND operation to combine the byte array of length being the length of the first integer value into a Simhash signature of the first text.

[0095] In one possible implementation, the hash algorithm is murmurhash, xxhash, or cityhash.

[0096] In one possible implementation, the vector addition instruction is based on the SIMD instruction set.

[0097] In one possible implementation, the SIMD instruction set is the AVX instruction set for x86 architecture microprocessors, the instruction set for PowerPC architecture microprocessors, the instruction set for SPARC architecture microprocessors, or the MSA instruction set for MIPS architecture microprocessors.

[0098] In one possible implementation, the length of the first integer value is 1 byte, 2 bytes, 4 bytes, or 8 bytes.

[0099] In one possible implementation, the first text comes from a webpage, a paper, or user feedback.

[0100] In one possible implementation, the Simhash signature of the first text is used for deduplication, deduplication, modeling, or statistics.

[0101] See Figure 8 , Figure 8 This is a schematic diagram of a Simhash calculation module provided in an embodiment of this application. Figure 8As shown, the Simhash calculation module 800 includes multiple units, including a word segmentation unit 810, a hash calculation unit 820, a weighted numeric string generation unit 830, an accumulation calculation unit 840, and a dimensionality reduction unit 850. The Simhash calculation module 800 receives input text 802 and outputs a corresponding Simhash signature 804. Here, the input text 802 can refer to any text among the first text, second text, or multiple texts used for text similarity calculation in the above method embodiments. The following description, in conjunction with the above method embodiments, further clarifies this. Figure 8 The internal operating mechanism of the Simhash calculation module 800.

[0102] The word segmentation unit 810 receives the input text 802 and performs word segmentation on the input text 802 to obtain a word segmentation result including at least one word, and assigns a word weight to each of the at least one word. When the input text 802 is a first text, the word segmentation unit 810 performs word segmentation on the first text to obtain a first word segmentation result, which includes at least one first word and a word weight for each first word. The word segmentation unit 810 transmits the word segmentation result of the input text 802 to the hash calculation unit 820. The hash calculation unit 820 calculates the hash value of each of the at least one word using a hash algorithm. The hash calculation unit 820 transmits the calculated hash value to the weighted number string generation unit 830. The weighted number string generation unit 830 also obtains the word weight of each of the at least one word from the word segmentation unit 810. The weighted number string generation unit 830 performs weighted calculation on the hash value and word weight of each word to obtain a weighted number string. When the input text 802 is the first text, the weighted numeric string generation unit 830 generates a weighted numeric string for each of the first words according to the hash value and word weight of the first word. The weighted numeric string generation unit 830 transmits the generated weighted numeric string to the accumulation calculation unit 840. The accumulation calculation unit 840 performs vectorized accelerated accumulation calculation on the weighted numeric string of each of the at least one word to obtain the sequence string of the input text 802. When the input text 802 is the first text, the accumulation calculation unit 840 performs vectorized accumulated calculation on the weighted numeric string of each of the first words to obtain the sequence string of the first text. The accumulation calculation unit 840 transmits the sequence string of the input text 802 to the dimensionality reduction unit 850. The dimensionality reduction unit 850 performs vectorized accelerated dimensionality reduction on the sequence string of the input text 802 to obtain the Simhash signature 804 of the input text 802. When the input text 802 is the first text, the dimensionality reduction unit 850 is used to perform vectorization dimensionality reduction on the sequence string of the first text to obtain the Simhash signature of the first text.

[0103] in, Figure 8 For details regarding the accumulation calculation unit 840 and the dimensionality reduction unit 850, please refer to the above method embodiments, including the above. Figure 6 and Figure 7 The details regarding the use of vector operation instructions to batch load hash weighted results to be accumulated, as well as batch data accumulation and batch data comparison, will not be elaborated here. Figure 8 The Simhash calculation module 800 shown here has implemented vectorization optimization to address the characteristics of the data flow generated in the step of accumulating the hash weighted results of all words in the Simhash-based text similarity calculation process, as well as the algorithmic characteristics of the Simhash-based text similarity calculation process, thus significantly improving the calculation speed. For the complex and varied computational needs of application scenarios requiring Simhash calculations on massive amounts of text such as web pages, papers, and user feedback, it also sets corresponding integer numerical lengths to reflect the impact of specific word segmentation strategies and NLP models on the data flow, thereby flexibly adapting to various application scenarios and improving overall performance.

[0104] It should be understood that, Figure 8 The Simhash calculation module 800 shown can be considered as a system. Figure 8 An embodiment provides a system comprising the various units of the Simhash calculation module 800 described above.

[0105] See Figure 9 , Figure 9 Provided for the embodiments of this application Figure 8 A schematic diagram of the accumulation calculation unit of the Simhash calculation module. (See diagram below.) Figure 9 As shown, the accumulation calculation unit includes a first register 902, a second register 904, and an instruction controller 910. As described above, Figure 6 and Figure 7 The first register YMM0 and the second register YMM1 mentioned in the embodiments are for illustrative purposes only. In actual applications, the names of the registers used can be determined according to the specific instruction set used, and the compiler can specify the specific registers to be used. Therefore, as long as the operating principle mentioned in the embodiments of this application is satisfied, any register or register name specified by the instruction set or the compiler can be used to correspond to the aforementioned first register YMM0 and / or second register YMM1. Figure 9In this embodiment, the first register 902 may correspond to the first register YMM0, and the second register 904 may correspond to the second register YMM1; or, the first register 902 may correspond to the second register YMM1, and the second register 904 may correspond to the first register YMM0. The instruction controller 910 is used to control the first register 902 and the second register 904 according to the instruction set for support vector operations, such as the AVX instruction set of the x86 architecture microprocessor, thereby implementing the above-mentioned operations related to the first register YMM0 and the second register YMM1.

[0106] See Figure 8 and Figure 9 This application provides a system. The system includes: a word segmentation unit, configured to segment a first text into words to obtain a first word segmentation result including at least one word, and to assign a word weight to each of the at least one word; a hash calculation unit, configured to calculate the hash value of each word; a weighted numeric string generation unit, configured to generate a weighted numeric string for each word according to the hash value and word weight; an accumulation calculation unit, configured to perform vectorized accumulation calculation on the weighted numeric strings of each word to obtain a sequence string of the first text; and a dimensionality reduction unit, configured to perform vectorized dimensionality reduction on the sequence string of the first text to obtain a first Simhash signature of the first text.

[0107] In one possible implementation, the accumulation calculation unit includes a first register and a second register. The accumulation calculation unit performs vectorized accelerated accumulation calculation to obtain a sequence string of the first text, including: traversing the weighted numeric string of each of the at least one word segment, loading the byte array corresponding to a specific number of weighted numeric strings into the second register each time, and performing batch accumulation calculation on the weighted numeric strings stored in the first register and the second register respectively through vector addition instructions to obtain the accumulation calculation result, and then storing the accumulation calculation result into the first register.

[0108] In one possible implementation, the specific number is based on the length of a first integer value and the width of the second register, which is equal to the width of the first register.

[0109] In one possible implementation, the first integer numerical length is based on the first word segmentation result of the first text, and the length of the byte array corresponding to the weighted number string of each of the at least one word is the first integer numerical length.

[0110] In one possible implementation, the first register is initialized with the length of the first integer value at least before the vectorized accelerated accumulation calculation is performed.

[0111] In one possible implementation, the vector addition instruction is based on the length of the first integer value.

[0112] In one possible implementation, the second register is initialized according to the length of the first integer value after the vectorized accelerated accumulation calculation and before the vectorized accelerated dimensionality reduction of the sequence string of the first text.

[0113] See Figure 10 , Figure 10 This is a schematic diagram of a computing device 1000 provided in an embodiment of this application. The computing device 1000 includes one or more processors 1010, a communication interface 1020, and a memory 1030. The processors 1010, communication interface 1020, and memory 1030 are interconnected via a bus 1040. Optionally, the computing device 1000 may further include an input / output interface 1050, which is connected to input / output devices for receiving user-set parameters, etc. The computing device 1000 can be used to implement some or all of the functions of the device embodiment or system embodiment in the above-described embodiments of this application; the processor 1010 can also be used to implement some or all of the operation steps of the method embodiment in the above-described embodiments of this application. For example, the specific implementation of various operations performed by the computing device 1000 can be referred to the specific details in the above embodiments, such as the processor 1010 being used to execute some or all of the steps or operations in the above-described method embodiments. For example, in the embodiments of this application, the computing device 1000 can be used to implement some or all of the functions of one or more components in the above-described device embodiments. In addition, the communication interface 1020 can be used for communication functions necessary to implement the functions of these devices and components, and the processor 1010 can be used for processing functions necessary to implement the functions of these devices and components.

[0114] It should be understood that, Figure 10 The computing device 1000 may include one or more processors 1010, and the multiple processors 1010 may collaboratively provide processing power in a parallel connection mode, a serial connection mode, a serial-parallel connection mode, or an arbitrary connection mode; or the multiple processors 1010 may form a processor sequence or a processor array; or the multiple processors 1010 may be divided into a main processor and an auxiliary processor; or the multiple processors 1010 may have different architectures, such as adopting a heterogeneous computing architecture. Furthermore, Figure 10 The structural and functional descriptions of the computing device 1000 shown are exemplary and non-limiting. In some exemplary embodiments, the computing device 1000 may include... Figure 10The diagram shows more or fewer components, or combinations of some components, or splitting of some components, or different arrangements of components.

[0115] The processor 1010 can have various specific implementations. For example, the processor 1010 may include one or more combinations of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or a data processing unit (DPU). This application embodiment does not impose specific limitations. The processor 1010 can also be a single-core processor or a multi-core processor. The processor 1010 can be a combination of a CPU and hardware chips. The aforementioned hardware chips can be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The aforementioned PLDs can be complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), generic array logic (GALs), or any combination thereof. The processor 1010 can also be implemented independently using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs). The communication interface 1020 can be a wired interface or a wireless interface for communicating with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., while the wireless interface can be a cellular network interface or a wireless LAN interface.

[0116] Memory 1030 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 1030 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM). The memory 1030 can also be used to store program code and data, so that the processor 1010 can call the program code stored in the memory 1030 to execute some or all of the operation steps in the above method embodiments, or to execute the corresponding functions in the above device embodiments. Furthermore, the computing device 1000 may include, compared to... Figure 10 The number of components displayed may be more or less, or there may be different component configurations.

[0117] The 1040 bus can be a peripheral component interconnect (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The 1040 bus can be divided into address bus, data bus, control bus, etc. In addition to the data bus, the 1040 bus can also include a power bus, control bus, and status signal bus. However, for clarity, Figure 10 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0118] The methods and devices provided in this application are based on the same inventive concept. Since the principles by which the methods and devices solve problems are similar, the embodiments, implementation methods, examples, or methods of implementation of the methods and devices can be referred to each other, and repeated details will not be repeated. This application also provides a system comprising multiple computing devices, the structure of each computing device of which can refer to the structure of the computing devices described above. The functions or operations achievable by this system can refer to the specific implementation steps in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0119] This application also provides a computer-readable storage medium storing computer instructions. When these computer instructions are executed on a computing device (such as one or more processors), they can implement the method steps described in the above method embodiments. The specific implementation of the above method steps by the processor of the computer-readable storage medium can refer to the specific operations described in the above method embodiments and / or the specific functions described in the above device embodiments, and will not be repeated here.

[0120] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. This application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Embodiments of this application can be implemented wholly or partially by software, hardware, firmware, or any other combination. When implemented in software, the above embodiments can be implemented wholly or partially as a computer program product. This application can take the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program code. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. Computer-readable storage media can be any available medium that a computer can access, or a data storage device such as a server or data center that contains one or more sets of available media. Available media can be magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media, or semiconductor media. Semiconductor media can be solid-state drives, random access memory, flash memory, read-only memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, registers, or any other suitable form of storage medium.

[0121] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. Each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0122] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a certain embodiment can be referred to in the relevant descriptions of other embodiments. Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. The steps in the methods of the embodiments of this application can be adjusted in order, combined, or deleted according to actual needs; the modules in the systems of the embodiments of this application can be divided, combined, or deleted according to actual needs. If these modifications and variations of the embodiments of this application fall within the scope of the claims of this application and their equivalents, then this application also intends to include these modifications and variations.

Claims

1. A method for calculating text similarity, characterized in that, The method includes: The first text is segmented to obtain a first segmentation result, which includes at least one first segment and the word weight of each first segment; For each of the first word segments, a weighted numeric string is generated based on the hash value and word weight of the first word segment; and For each weighted numeric string of the first word segment, vectorized accumulation calculation is performed to obtain the sequence string of the first text, and vectorized dimensionality reduction is performed on the sequence string of the first text to obtain the Simhash signature of the first text. Obtain the Simhash signature of the second text, and based on the Simhash signature of the first text and the Simhash signature of the second text, obtain the similarity between the first text and the second text; The step of performing vectorized accelerated accumulation calculation to obtain the sequence string of the first text includes: traversing the weighted number string of each of the at least one word segment, loading the byte array corresponding to a specific number of weighted number strings into the second register each time, and performing batch accumulation calculation on the weighted number strings stored in the first register and the second register respectively through vector addition instructions to obtain the accumulation calculation result, and then storing the accumulation calculation result in the first register. The specific number is determined based on the length of a first integer value and the width of the second register. The first integer length is determined based on the first word segmentation result of the first text. The first register is initialized according to the first integer length at least before the vectorized accumulation calculation is performed. The second register is initialized according to the first integer length after the vectorized accelerated accumulation calculation is performed and before the vectorized dimensionality reduction of the sequence string of the first text is performed.

2. The method according to claim 1, characterized in that, The length of the byte array corresponding to each of the weighted numeric strings of the at least one word is the length of the first integer value.

3. The method according to claim 2, characterized in that, The vectorization and dimensionality reduction of the sequence string of the first text includes: The first register and the second register are compared in batches using a batch comparison instruction to obtain a comparison result, and the comparison result is stored in the first register.

4. The method according to claim 3, characterized in that, The step of vectorizing and reducing the dimensionality of the sequence string of the first text further includes: The storage result of the first register is saved to a byte array of the same length as the first integer value to generate the first Simhash signature of the first text.

5. The method according to claim 4, characterized in that, The step of vectorizing and reducing the dimensionality of the sequence string of the first text further includes: The first Simhash signature of the first text is formed by combining the byte array of the first integer value length using left shift and logical AND operation.

6. The method according to any one of claims 1 to 5, characterized in that, The vector addition instruction is based on the SIMD instruction set.

7. The method according to claim 6, characterized in that, The SIMD instruction set is the AVX instruction set for x86 architecture microprocessors, the instruction set for PowerPC architecture microprocessors, the instruction set for SPARC architecture microprocessors, or the MSA instruction set for MIPS architecture microprocessors.

8. A computing device, characterized in that, The computing device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method according to any one of claims 1 to 7.

Images