Address class short text similarity calculation method and device, equipment and medium

Abstract

The application provides an address type short text similarity calculation method and device, equipment and medium, including obtaining a Chinese address type short text, dividing the Chinese address type short text into a text character array and a number character array according to text features, calculating the similarity of two Chinese address type short texts based on the text character array and the number character array corresponding to the two Chinese address type short texts, realizing the quantification of similarity and efficient batch processing, and improving the efficiency and accuracy of security checks. Efficient batch processing can be realized through server basic hardware, which reduces resource consumption while ensuring the accuracy of similarity calculation, thereby providing technical support for sensitive information identification in the security check scene.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of text processing, and particularly relates to an address type short text similarity calculation method, device, equipment and medium. BACKGROUND

[0002] In many application scenarios, address type text data similarity identification is one of the core links of sensitive information detection. Due to the high randomness of address type text data in the input process, such as field missing (for example, "certain district" is abbreviated as "certain district"), format inconsistency, and case mixing (for example, "1 building" and "one building"), the same address may exist in multiple forms. In addition, the security check needs to compare a large amount of address type text data in batches, which poses a double challenge to the computing efficiency and hardware resources.

[0003] At present, there are many methods for realizing text similarity calculation technology, but there are few special optimization algorithms for address type text features. In addition, many algorithms have high time complexity or many NLP models need to rely on the support of GPU, so most of them face the contradiction between computing efficiency and resource consumption, high preprocessing dependence, insufficient batch processing capacity and other problems. SUMMARY

[0004] In view of the above problems existing in the prior art, the application provides an address type short text similarity calculation method, device, equipment and medium.

[0005] To achieve the above purpose, the technical scheme adopted by the application is as follows:

[0006] The application provides an address type short text similarity calculation method, which comprises the following steps:

[0007] Obtaining Chinese address type short texts;

[0008] Segmenting and grouping Chinese characters and numbers in each Chinese address type short text to obtain text character arrays and number character arrays corresponding to each Chinese address type short text;

[0009] Calculating the similarity of two Chinese address type short texts based on the text character arrays and number character arrays corresponding to the two Chinese address type short texts.

[0010] Preferably, the similarity of two Chinese address type short texts is calculated based on the text character arrays and number character arrays corresponding to the two Chinese address type short texts, which comprises the following steps:

[0011] Calculating the similarity score of the text character arrays corresponding to the two Chinese address type short texts;

[0012] Calculating the similarity score of the number character arrays corresponding to the two Chinese address type short texts;

[0013] The similarity scores of the text character arrays and the numerical character arrays corresponding to the two Chinese address type short texts are fused by weighting to obtain a comprehensive similarity of the two Chinese address type short texts.

[0014] Preferably, a similarity threshold is further set, and if the comprehensive similarity of the two Chinese address type short texts is greater than the set threshold, the two Chinese address type short texts are marked as a high similarity pair.

[0015] In another aspect, an address type short text similarity calculation device is provided, comprising:

[0016] A first module is configured to obtain Chinese address type short texts.

[0017] A second module is configured to segment and group Chinese characters and numbers in each Chinese address type short text to obtain text character arrays and numerical character arrays corresponding to each Chinese address type short text.

[0018] A third module is configured to calculate the similarity of two Chinese address type short texts based on the text character arrays and the numerical character arrays corresponding to the two Chinese address type short texts.

[0019] In another aspect, the present application provides a computer device comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps of the address type short text similarity calculation method when executing the computer program.

[0020] In another aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the address type short text similarity calculation method when executed by a processor.

[0021] In another aspect, the present application provides a computer program product, which is stored on a computer readable storage medium and includes computer instructions, which make a computer device implement the steps of the address type short text similarity calculation method when executed by a processor.

[0022] Compared with the prior art, the technical effects of the present application are as follows:

[0023] The application provides an address type short text similarity calculation method, acquires Chinese address type short texts, divides the Chinese address type short texts into text character arrays and digital character arrays according to text features, calculates the similarity of two Chinese address type short texts based on the text character arrays and the digital character arrays corresponding to the two Chinese address type short texts, realizes the quantification of similarity and efficient batch processing, and improves the efficiency and accuracy of security checks. Efficient batch processing can be realized through server basic hardware (regular memory and CPU), which reduces resource consumption while ensuring the accuracy of similarity calculation, thereby providing technical support for sensitive information identification in the security check scenario. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the structures shown in the drawings without creative labor.

[0025] Figure 1 A flowchart of an address type short text similarity calculation method provided by an embodiment;

[0026] Figure 2 A flowchart of similarity calculation for two character arrays provided by an embodiment.

[0027] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0029] REFERENCE Figure 1 In an embodiment, an address type short text similarity calculation method is provided, comprising:

[0030] acquiring Chinese address type short texts;

[0031] segmenting and grouping Chinese characters and numbers in each Chinese address type short text to obtain text character arrays and digital character arrays corresponding to each Chinese address type short text;

[0032] The similarity of two Chinese address type short texts is calculated based on text character arrays and number character arrays corresponding to the two Chinese address type short texts.

[0033] In practical applications, for a large batch of acquired Chinese address type short texts, the length of each Chinese address type short text can be first arranged, and then the Chinese address type short texts are sorted according to the length, wherein the length arrangement includes removing province, city, district, county and other illegal characters, and then the lengths of the Chinese address type short texts after length arrangement are arranged in descending order (or the explicit standard address is placed at the first position of the set). In this way, when subsequent similarity calculation is performed, a multi-thread parallel computing framework can be enabled according to the sorting result, the first Chinese address type short text is calculated with the Chinese address type short texts sorted after it, the second Chinese address type short text is calculated with the Chinese address type short texts sorted after it, and so on, to complete the similarity calculation and labeling of all Chinese address type short texts.

[0034] The present application realizes the rapid quantitative calculation and batch processing of short address text similarity with low space-time complexity, significantly improves the accuracy and execution efficiency of the security check. The present application only needs the conventional server memory and hard disk resources, does not need special hardware acceleration equipment, and has wide deployment ability. It solves the variant matching (such as abbreviation, format disorder) and real-time batch processing demand of a large amount of short address data in the security check, and provides effective technical support for sensitive information identification.

[0035] In an embodiment, based on the importance of the number text in the Chinese address type short text, the text characters and number characters of the Chinese address type short text are segmented and grouped, and then the similarity of two Chinese address type short texts is calculated based on the text character arrays and number character (including English) arrays corresponding to the two Chinese address type short texts, including:

[0036] Text group collision: calculating the similarity score of the text character arrays corresponding to the two Chinese address type short texts;

[0037] Number group collision: calculating the similarity score of the number character arrays corresponding to the two Chinese address type short texts;

[0038] The similarity score of the text character arrays and the similarity score of the number character arrays corresponding to the two Chinese address type short texts are weighted and fused to obtain the comprehensive similarity of the two Chinese address type short texts.

[0039] The present application only needs the conventional server memory and hard disk resources, does not need special hardware acceleration equipment, and has wide deployment ability, while significantly reducing the consumption of computing resources (time / space) and ensuring the accuracy of similarity determination;

[0040] Further, a similarity threshold is set, and if the comprehensive similarity of the two Chinese address type short texts is greater than the set threshold, the two Chinese address type short texts are marked as a high similarity pair.

[0041] The comprehensive similarity of the two Chinese address type short texts is:

[0042]

[0043] wherein, A Similarity represents the comprehensive similarity of the two Chinese address type short texts, A text represents the similarity score of the text character array corresponding to the two Chinese address type short texts, and A Num represents the similarity score of the numerical character array corresponding to the two Chinese address type short texts.

[0044] In an embodiment, a character array similarity calculation method is created, and then the created character array similarity calculation method is called to calculate the similarity score of the text character array corresponding to the two Chinese address type short texts and the similarity score of the numerical character array corresponding to the two Chinese address type short texts.

[0045] Specifically, the character array similarity calculation method comprises:

[0046] Input two character arrays G_a and G_b to be calculated for similarity, wherein the two character arrays G_a and G_b are text character arrays or numerical character arrays.

[0047] According to the lengths of the two character arrays G_a and G_b, calculate the theoretical maximum score T max when the two character arrays G_a and G_b are completely the same.

[0048] Double loop collision matching combined with a sliding window mechanism is used to calculate the actual matching score T actual of the two character arrays G_a and G_b.

[0049] The actual matching score is divided by the theoretical maximum score to obtain a normalized similarity score, i.e.

[0050] Based on the above method, the similarity between all pairs of Chinese address type short texts is obtained, and according to the set similarity threshold, the labeled data is grouped and output, and the high similarity Chinese address type short text pairs and the dissimilar Chinese address type short text pairs are output. Specifically, refer to Figure 2 , which comprises:

[0051] Input the text character array and the numerical character array corresponding to the two Chinese address type short texts.

[0052] ​​The character array similarity calculation method is called to complete text group similarity calculation, including calculating the theoretical maximum score of the text character arrays corresponding to the two Chinese address type short texts, calculating the actual matching score of the text character arrays corresponding to the two Chinese address type short texts, and obtaining the normalized similarity score of the text character arrays corresponding to the two Chinese address type short texts by dividing the actual matching score of the text character arrays corresponding to the two Chinese address type short texts by the theoretical maximum score;

[0053] The character array similarity calculation method is called to complete text group similarity calculation, including calculating the theoretical maximum score of the text character arrays corresponding to the two Chinese address type short texts, calculating the actual matching score of the text character arrays corresponding to the two Chinese address type short texts, and obtaining the normalized similarity score of the text character arrays corresponding to the two Chinese address type short texts by dividing the actual matching score of the text character arrays corresponding to the two Chinese address type short texts by the theoretical maximum score;

[0054] The normalized similarity score of the text character arrays corresponding to the two Chinese address type short texts and the normalized similarity score of the digital character arrays are fused to obtain the comprehensive similarity of the two Chinese address type short texts.

[0055] According to the similarity threshold, data labeling is completed: if the comprehensive similarity of the two Chinese address type short texts is greater than the set threshold, the two Chinese address type short texts are marked as a high similarity pair, otherwise they are not similar.

[0056] The actual matching score of the two character arrays G_a and G_b is calculated through the characteristics of the Chinese address type short text and the dynamic sliding window matching mechanism, which is suitable for the rapid deduplication and sensitive information identification of massive unstructured address data in the security check, and relies on the general server memory and CPU resources, so as to realize the most efficient discovery and finding of similar addresses for merging and processing in the security check process.

[0057] The theoretical maximum score of the two character arrays G_a and G_b being completely identical is calculated according to the length of the two character arrays G_a and G_b;

[0058] The theoretical maximum score T of the two character arrays G_a and G_b being completely identical max , the calculation formula is:

[0059] ;

[0060] Wherein: represents the length of the array G_a, and the length of the array G_a and G_b is the same when the array G_a and G_b are completely identical; represents the start position reward function, and for the two character arrays G_a and G_b, if the two characters at the start position are the same, a set reward score (such as 1 point or other scores) is returned, otherwise 0 is returned. δend(G_a, G_b) represents the end position reward score function, which returns a set reward score (e.g. 1 or other score) if the last two characters of the end positions of the two character arrays G_a and G_b are the same, otherwise returns 0.

[0061] Further, the actual matching score T of the two character arrays G_a and G_b is calculated actual , including:

[0062] (1) Calculate the start position reward score: if the last w characters of the array G_a are the same as the last w characters of the array G_b, then obtain the corresponding set reward score according to the start position reward score function δstart(G_a, G_b); , where , is the length of the array G_a after removing the sliding window length w at the end, and the corresponding set reward score is obtained according to the start position reward score function δstart(G_a, G_b);

[0063] (2) Calculate the end position reward score: if the last character of the end position of the array G_a is the same as the last character of the end position of the array G_b, then obtain the corresponding set reward score according to the end position reward score function δend(G_a, G_b);

[0064] (3) Initialize the score array Sa[i] with a length equal to the array G_a, where ;

[0065] (4) Double loop collision matching, outer loop for each character of the array G_a, and inner loop for each character of the array G_b to perform collision scoring:

[0066] a. Outer loop: initialize the temporary score array Score[j] with a length equal to the array G_b, where , and , is the length of the array G_b;

[0067] b. Inner loop: compare the character G_a[i] of the outer array G_a with the character G_b[j] of the inner array G_b, if they are equal, then Score[j] records 1, if they are not equal, then Score[j] records 0; if w consecutive 0s are found during the comparison, then the inner loop is exited, otherwise the comparison is continued until j = L G_b ; After the inner loop is completed, the sum of Score[j] is calculated and assigned to the score array Sa[i], i.e.

[0068] ;

[0069] (5) Sum , and combine the start position reward score and the end position reward score to obtain the actual matching score T of the two character arrays G_a and G_b: actual :

[0070] ;

[0071] Wherein the starting position reward points of step (1) are calculated by removing the sliding window length w at the end of the array G_a, and if the length of the array G_a is not enough, no points are counted. The purpose of removing w is to ensure that only the matching within the effective length w is given the set reward points.

[0072] Finally, the normalized similarity is calculated: the actual matching score is divided by the theoretical maximum score to obtain the normalized similarity score, so as to achieve the goal of accurate evaluation, and the formula is:

[0073] ；

[0074] In the above embodiment, the score array Sa[i] records the similarity score of each substring of the array G_a from the i position to L G_a end, and the array G_b.

[0075] The present application realizes the quantification of similarity and efficient batch processing by grouping, sorting, multiple scoring and other steps of address-specific text features, and the collision comparison algorithm of address similarity, improves the efficiency and accuracy of security check. Through the server basic hardware (conventional memory and CPU), efficient batch processing can be realized, which reduces resource consumption while ensuring the accuracy of similarity calculation, thereby providing technical support for sensitive information identification in the security check scene.

[0076] On the other hand, an address class short text similarity calculation device is provided, comprising:

[0077] The first module is used for acquiring Chinese address class short texts;

[0078] The second module is used for segmenting and grouping Chinese characters and numbers in each Chinese address class short text respectively to obtain text character arrays and number character arrays corresponding to each Chinese address class short text;

[0079] The third module is used for calculating the similarity of two Chinese address class short texts based on the text character arrays and number character arrays corresponding to the two Chinese address class short texts.

[0080] In another aspect, the present application provides a computer device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the address type short text similarity calculation method provided in any of the above embodiments when executing the computer program. The computer device can be a server. The computer device comprises a processor, a memory, a network interface and a database connected by a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store sample data. The network interface of the computer device is configured to communicate with an external terminal through a network connection.

[0081] In another aspect, the present application provides a computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the steps of the address type short text similarity calculation method provided in any of the above embodiments.

[0082] A person of ordinary skill in the art can understand that all or part of the processes in the above embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium and executed to include the processes of the above embodiments. Any reference to memory, storage, database or other medium in the embodiments of the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0083] The details of the present application are described above.

[0084] Any technical features in the above-described embodiments can be combined in any manner, and for the sake of brevity, not all possible combinations are described, however, any combination of the technical features is considered to be within the scope of the present specification.

[0085] The above-described embodiments are merely illustrative for the present application and are not used to limit the present application. It should be pointed out that, for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these should be included in the protection scope of the present application.

[0086] The above-described embodiments are merely illustrative for the present application and are not used to limit the present application. It should be pointed out that, for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these should be included in the protection scope of the present application. The above-described embodiments are merely illustrative for the present application and are not used to limit the present application. It should be pointed out that, for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these should be included in the protection scope of the present application.

Claims

1. A method for calculating the similarity of short address texts, characterized in that, include: Retrieve short text of Chinese address data; The Chinese characters and numbers in each Chinese address short text are divided into groups to obtain the text character array and numeric character array corresponding to each Chinese address short text. The similarity between two short Chinese address texts is calculated based on their corresponding text character arrays and numeric character arrays, including: Create a method for calculating the similarity of character arrays, including: Input two character arrays G_a and G_b to be used for similarity calculation. The two character arrays G_a and G_b can be text character arrays or numeric character arrays. Calculate the theoretical maximum score when the two character arrays G_a and G_b are exactly the same, based on their lengths. Double-loop collision matching, combined with a sliding window mechanism, calculates the actual matching scores of the two character arrays G_a and G_b; The normalized similarity score is obtained by dividing the actual matching score by the theoretical maximum score. The similarity scores of the text character arrays corresponding to the two short Chinese address texts and the similarity scores of the numeric character arrays corresponding to the two short Chinese address texts are calculated by calling a pre-created character array similarity calculation method. The similarity scores of the text character arrays and the similarity scores of the numeric character arrays corresponding to the two short Chinese address texts are weighted and fused to obtain the comprehensive similarity between the two short Chinese address texts.

2. The method for calculating the similarity of short address texts according to claim 1, characterized in that, It also includes setting a similarity threshold. If the overall similarity of two short Chinese address texts is greater than the set threshold, then the two short Chinese address texts are marked as a highly similar pair.

3. The method for calculating the similarity of short address texts according to claim 1 or 2, characterized in that, The overall similarity between the two short Chinese address texts is: A Similarity =A text +2×A Num Among them, A Similarity A represents the overall similarity between two short texts of Chinese address types. text A represents the similarity score of the text character arrays corresponding to two short Chinese address texts. Num This represents the similarity score of numeric character arrays corresponding to two short texts of Chinese address classes.

4. The method for calculating the similarity of short address texts according to claim 3, characterized in that, Calculate the theoretical maximum score T when the two character arrays G_a and G_b are identical, based on their lengths. max The calculation formula is: in: This indicates the length of array G_a. If arrays G_a and G_b are exactly the same, then arrays G_a and G_b also have the same length. This function represents the reward score for the starting position. For two character arrays G_a and G_b, if the two characters at the starting positions are the same, it returns the set reward score; otherwise, it returns 0. This function represents the reward score for the last position. For two character arrays G_a and G_b, if the last two characters are the same, it returns the set reward score; otherwise, it returns 0.

5. The method for calculating the similarity of short address texts according to claim 3, characterized in that, Calculate the actual matching score T of the two character arrays G_a and G_b. actual ,include: (1) Calculate the starting position reward score: After removing the sliding window length w from the end of array G_a, if G_b[1]=G_a[i], where i∈[1,L] G_a-w ], L G_a-w Given the length of the array G_a after removing the sliding window length w from the end, the corresponding reward score is obtained according to the reward score function δstart() based on the starting position. (2) Calculate the end position bonus: If the last character of the last position of array G_a is the same as the last character of the last position of array G_b, then the corresponding bonus is obtained according to the end position bonus function δend(); (3) Initialize a score array Sa[i] of length equal to that of array G_a, where i∈[1,L] G_a ]; (4) Double loop collision matching: each character in the outer loop array G_a, and each character in the inner loop array G_b are used for collision scoring. a. Outer loop: Initialize a temporary score array Score[j] of length equal to array G_b, where j∈[1,L]. G_b ], L G_b The length of array G_b; b. Inner loop: Compare the character G_a[i] of the outer array G_a with the character G_b[j] of the inner array G_b. If they are equal, Score[j] is scored as 1 point; otherwise, Score[j] is scored as 0 points. If w consecutive zero scores appear during the comparison, the inner loop is exited; otherwise, the comparison continues until j=L. G_b When the inner loop completes, it sums up Score[j] and assigns the sum to the score array Sa[i]. (5) Summing the scores and combining them with the bonus points for the starting and ending positions, we obtain the actual matching score T for the two character arrays G_a and G_b. actual : 。 6. A device for calculating the similarity of short address texts, characterized in that, include: The first module is used to obtain short text of Chinese address formats; The second module is used to segment and group the Chinese characters and numbers in each Chinese address short text to obtain the text character array and numeric character array corresponding to each Chinese address short text. The third module is used to calculate the similarity between two short Chinese address texts based on their corresponding text character arrays and numeric character arrays, including: Create a method for calculating the similarity of character arrays, including: Input two character arrays G_a and G_b to be used for similarity calculation. The two character arrays G_a and G_b can be text character arrays or numeric character arrays. Calculate the theoretical maximum score when the two character arrays G_a and G_b are exactly the same, based on their lengths. Double-loop collision matching, combined with a sliding window mechanism, calculates the actual matching scores of the two character arrays G_a and G_b; The normalized similarity score is obtained by dividing the actual matching score by the theoretical maximum score. The similarity scores of the text character arrays corresponding to the two short Chinese address texts and the similarity scores of the numeric character arrays corresponding to the two short Chinese address texts are calculated by calling a pre-created character array similarity calculation method. The similarity scores of the text character arrays and the similarity scores of the numeric character arrays corresponding to the two short Chinese address texts are weighted and fused to obtain the comprehensive similarity between the two short Chinese address texts.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, The processor executes the steps of the address-based short text similarity calculation method as described in claim 1 when executing a computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the steps of the address-based short text similarity calculation method as described in claim 1.

Images