Similar address screening method, device, computing device and storage medium
By partitioning and vector matrix segmentation of address data, the similarity in address partition is calculated, batch screening of similar addresses is realized, and the problems of low computing efficiency and large resource occupation in the existing technology are solved, and processing efficiency and saving are improved. The cost.
Patent Information
- Application Number
- CN202110882531.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-08-02
AI Technical Summary
In the prior art, address similarity calculation efficiency is low, the operating cost is large, and the resources are occupied, so batch processing and batch screening of similar addresses cannot be realized.
By obtaining multiple address data, partitioning according to preset rules, generating the first address vector matrix corresponding to the address partition, and dividing it into multiple target vector matrices, and calculating the various address data in the address partition based on these matrices Similarity, filter out the target similar address data.
It realizes batch processing and batch screening of a large number of addresses, improves computing efficiency, and reduces resource occupation and operation costs.
Smart Images

Figure CN113626730B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a similar address screening method, device, computing equipment and storage medium. Background Art
[0002] With the rapid development of the Internet, the application of address data in express logistics and other industries is becoming more and more extensive. In the application process of massive address data, it is often necessary to calculate the similarity between addresses to determine the accuracy of address data.
[0003] At present, traditional address similarity is mostly calculated based on the relationship between two specific addresses, which cannot realize batch processing and batch screening of similar addresses. It has the problems of low computing efficiency, high operating cost and high resource consumption. Summary of the invention
[0004] The embodiments of the present invention provide a similar address screening method, apparatus, computing device and storage medium to solve the problems of low address similarity calculation efficiency, high operating cost and high resource occupation in the prior art.
[0005] In order to solve the above problems, the embodiment of the present invention is implemented as follows:
[0006] In a first aspect, an embodiment of the present invention discloses a similar address screening method, comprising:
[0007] Acquire multiple address data, and partition the address data according to a preset rule to obtain multiple address partitions;
[0008] For each address partition, based on the address data in the address partition, generate a first address vector matrix corresponding to the address partition;
[0009] Dividing the first address vector matrix to obtain a plurality of first target vector matrices;
[0010] Based on the multiple first target vector matrices, determining the similarity between each address data in the address partition;
[0011] Based on the similarity, target similar address data is screened out.
[0012] Optionally, partitioning the address data according to a preset rule includes:
[0013] Segmenting the address data to obtain an administrative region field and a specific address field of the address data;
[0014] According to the administrative region field, the address data with the same administrative region field are divided into the same address partition.
[0015] Optionally, generating a first address vector matrix corresponding to the address partition based on the address data in the address partition includes:
[0016] In the case where high-frequency characters exist in the address data, extracting the high-frequency characters in the address data as target address data; the high-frequency characters are address characters whose frequency of appearance in the address partition is higher than a preset frequency threshold;
[0017] Adding the target address data to the target word library;
[0018] The address vector corresponding to each target address data in the target vocabulary is generated by a preset vector generation model to obtain the first address vector matrix.
[0019] Optionally, determining the similarity between each address data in the address partition includes:
[0020] Copying the first address vector matrix to obtain a second address vector matrix;
[0021] Dividing the second address vector matrix to obtain a plurality of second target vector matrices; the second target vector matrix has the same number of rows and columns as the first target vector matrix;
[0022] For any one of the first target vector matrix and any one of the second target vector matrix, determine the position of the calculation result of the first target vector matrix and the second target vector matrix in the result matrix;
[0023] If the calculation result is in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are input into the preset similarity model for calculation to obtain a result sub-matrix; the upper triangular position is a position in the result matrix where the number of rows is not greater than the number of columns;
[0024] If the calculation result is not in the upper triangle position of the result matrix, the first target vector matrix and the second target vector matrix are not calculated.
[0025] Optionally, the filtering out target similar address data based on the similarity includes:
[0026] For each result sub-matrix, determine whether each element in the result sub-matrix is greater than a preset similarity threshold; each element in the result sub-matrix is an address similarity;
[0027] When the element is greater than a preset similarity threshold, save the element and the row and column values of the similarity in the result submatrix;
[0028] Determine the target address data corresponding to the element in the first address vector matrix based on the row and column values, and save the target address data and the element to obtain target similar address data;
[0029] If the element is not greater than the preset similarity threshold, the element is discarded.
[0030] In a second aspect, an embodiment of the present invention discloses a similar address screening device, comprising:
[0031] A partitioning module, used to obtain multiple address data, and partition the address data according to a preset rule to obtain multiple address partitions;
[0032] A generating module, configured to generate, for each address partition, a first address vector matrix corresponding to the address partition based on address data in the address partition;
[0033] A segmentation module, used for segmenting the first address vector matrix to obtain a plurality of first target vector matrices;
[0034] A determination module, configured to determine the similarity between each address data in the address partition based on the multiple first target vector matrices;
[0035] The screening module is used to screen out target similar address data based on the similarity.
[0036] Optionally, the partition module is specifically used to:
[0037] Segmenting the address data to obtain an administrative region field and a specific address field of the address data;
[0038] According to the administrative region field, the address data with the same administrative region field are divided into the same address partition.
[0039] Optionally, the generating module is specifically used for:
[0040] In the case where high-frequency characters exist in the address data, extracting the high-frequency characters in the address data as target address data; the high-frequency characters are address characters whose frequency of appearance in all address data in the address partition is higher than a preset frequency threshold;
[0041] Adding the target address data to the target word library;
[0042] The address vector corresponding to each target address data in the target vocabulary is generated by a preset vector generation model to obtain the first address vector matrix.
[0043] Optionally, the determining module is specifically used to:
[0044] Copying the first address vector matrix to obtain a second address vector matrix;
[0045] Dividing the second address vector matrix to obtain a plurality of second target vector matrices; the second target vector matrix has the same number of rows and columns as the first target vector matrix;
[0046] For any one of the first target vector matrix and any one of the second target vector matrix, determine the position of the calculation result of the first target vector matrix and the second target vector matrix in the result matrix;
[0047] If the calculation result is in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are input into the preset similarity model for calculation to obtain a result sub-matrix; the upper triangular position is a position in the result matrix where the number of rows is not greater than the number of columns;
[0048] If the calculation result is not in the upper triangle position of the result matrix, the first target vector matrix and the second target vector matrix are not calculated.
[0049] Optionally, the screening module is specifically used to:
[0050] For each result sub-matrix, determine whether each element in the result sub-matrix is greater than a preset similarity threshold; each element in the result sub-matrix is an address similarity;
[0051] When the element is greater than a preset similarity threshold, save the element and the row and column values of the similarity in the result submatrix;
[0052] Determine the target address data corresponding to the element in the first address vector matrix based on the row and column values, and save the target address data and the element to obtain target similar address data;
[0053] If the element is not greater than the preset similarity threshold, the element is discarded.
[0054] In a third aspect, an embodiment of the present invention further provides a computing device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the similar address screening method described in the first aspect.
[0055] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the similar address screening method described in the first aspect are implemented.
[0056] In an embodiment of the present invention, multiple address data are obtained, and the multiple address data are partitioned according to preset rules to obtain multiple address partitions; for each address partition, based on the address data in the address partition, a first address vector matrix corresponding to the address partition is generated; the first address vector matrix is segmented to obtain multiple first target vector matrices; based on the multiple first target vector matrices, the similarity between the respective address data in the address partition is determined; based on the similarity, the target similar address data is screened out. In an embodiment of the present invention, multiple address partitions are obtained by partitioning the address data, and the address vector matrix in each address partition is segmented and calculated at the same time, thereby realizing batch processing of a large number of addresses, being able to batch screen out addresses with high similarity, and since the calculation is performed on the segmented small matrix, less resources are occupied, the calculation efficiency is high, and the operation cost is saved. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 A flowchart showing the steps of a similar address screening method of the present invention is shown;
[0058] Figure 2 The processing result of a very large matrix in the prior art is shown;
[0059] Figure 3 A schematic diagram of a process of address data segmentation of the present invention is shown;
[0060] Figure 4 A schematic diagram of a generation process of a first address vector matrix of the present invention is shown;
[0061] Figure 5 A schematic diagram of a similarity calculation process of a first address vector matrix of the present invention is shown;
[0062] Figure 6 A schematic diagram of a processing flow of an address screening method of the present invention is shown;
[0063] Figure 7 A structural block diagram of a similar address screening device of the present invention is shown. DETAILED DESCRIPTION
[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0065] Reference Figure 1 , shows a flowchart of the steps of a similar address screening method of the present invention, which may specifically include:
[0066] Step 101: Acquire multiple address data, and partition the address data according to a preset rule to obtain multiple address partitions.
[0067] In the embodiment of the present invention, the address data may refer to data representing the address, such as a delivery address, etc. The preset rule may refer to a pre-set rule for partitioning the address data and obtaining address partitions, such as partitioning the address data according to provinces, cities, and districts, etc., which is not limited in the embodiment of the present invention.
[0068] The application scenario of the embodiment of the present invention is batch processing of a large number of addresses and batch screening of similar addresses. In this step, a large amount of address data can be read by using the computing engine spark. The spark is a distributed, fast and general computing engine designed for large-scale data processing. After obtaining the massive address data, the address data can be partitioned by Pyspark to obtain multiple address partitions. Among them, the Pyspark is a distributed computing framework combined with python and spark, and pyspark is a programming interface provided for python developers. Data development on spark can be realized through the python language.
[0069] It should be noted that in this step, each address partition obtained by partitioning the massive address data is divided into different tasks of Spark for calculation. In this way, by distributing the calculation of different address partitions in different task processes, the amount of data processing can be reduced and the processing efficiency can be improved.
[0070] Step 102: For each address partition, generate a first address vector matrix corresponding to the address partition based on the address data in the address partition.
[0071] In the embodiment of the present invention, the first address vector matrix may refer to an address vector matrix corresponding to the address data in the address partition, and the address vector matrix may include each address data in the address partition.
[0072] In this step, after multiple address partitions are obtained, the address data in each address partition may be vectorized to obtain a first address vector matrix composed of multiple address vectors. Subsequently, the similarity between the address data may be obtained based on matrix calculation.
[0073] Step 103: divide the first address vector matrix into multiple first target vector matrices.
[0074] In an embodiment of the present invention, the first target vector matrix may refer to a vector matrix obtained after segmenting the first address vector matrix. The size of the first target vector matrix may be set based on the processing capability of the computing device. The greater the processing capability of the computing device, the larger the first target vector matrix.
[0075] In this step, since a large amount of address data is obtained, the first address vector matrix may be a very large matrix. Directly processing the very large matrix will cause memory overflow and calculation cannot be performed. Figure 2 FIG. 2 shows the processing result of a very large matrix in the prior art. Figure 2 As shown in , if the calculation is performed directly on an extra-large matrix, the computing device will display that the system memory is exceeded and the calculation cannot be performed. In an embodiment of the present invention, the first address vector matrix can be segmented to obtain multiple small matrices that can be directly calculated, that is, multiple first target vector matrices. Specifically, the batch_size parameter in the custom segmentation function can be controlled to control the size of the first target vector matrix that participates in the subsequent calculation, so as to reduce the amount of data processing and achieve rapid data processing. Of course, in actual applications, other methods can also be used to segment the first address vector matrix, and the embodiment of the present invention is not limited to this.
[0076] Step 104: Determine the similarity between each address data in the address partition based on the multiple first target vector matrices.
[0077] In the embodiment of the present invention, the similarity may refer to the similarity between address data and address data. The similarity is a value between 0 and 1. The higher the similarity, the more similar the two addresses are.
[0078] In this step, after the first address vector matrix is divided into multiple first target vector matrices, the multiple first target vector matrices can be input into a preset similarity model in pairs, and the similarity between the address data in the address partition can be obtained by matrix calculation. The preset similarity model can be constructed based on the similarity formula calculated by the vector cosine distance, or based on other similarity calculation methods, which is not limited in the embodiment of the present invention.
[0079] Step 105: Filter out target similar address data based on the similarity.
[0080] In the embodiment of the present invention, the target similar address data may refer to address data with a similarity higher than a preset similarity threshold. After calculating the similarity between the respective address data, it may be determined whether the similarity is higher than the preset similarity threshold, and subsequently the address data with a similarity higher than the preset similarity threshold may be screened out as the target similar address data, thereby realizing batch screening of similar addresses.
[0081] In summary, a similar address screening method provided by an embodiment of the present invention obtains multiple address data, and partitions the multiple address data according to preset rules to obtain multiple address partitions; for each address partition, based on the address data in the address partition, generates a first address vector matrix corresponding to the address partition; divides the address vector matrix to obtain multiple first target vector matrices; based on the multiple first target vector matrices, determines the similarity between each address data in the address partition; based on the similarity, screens out target similar address data. In an embodiment of the present invention, multiple address partitions are obtained by partitioning the address data, and the address vector matrix in each address partition is divided and calculated at the same time, thereby realizing batch processing of a large number of addresses, and being able to batch screen out addresses with high similarity, and because the calculation is performed on the divided small matrix, less resource occupancy is achieved, the calculation efficiency is high, and the operating cost is saved.
[0082] Optionally, in the embodiment of the present invention, step 101 may specifically include the following steps 1011 to 1012:
[0083] Step 1011: Segment the address data to obtain the administrative region field and the specific address field of the address data.
[0084] In the embodiment of the present invention, the administrative region field may refer to a field representing an administrative region in the address data, such as a province, city, district, etc. The specific address field may refer to the remaining fields in the address data except for the province, city, district, etc., such as a street, community, building number, etc.
[0085] In this step, after obtaining the original massive address data by using spark, the original address data is first preprocessed, and the preprocessing may include deduplication and cleaning. By preprocessing the address data, invalid and duplicate address data can be deleted to reduce the amount of data processing. Afterwards, the address data can be segmented by the cpca module to obtain the administrative district field and specific address field of the address data. The segmentation tool cpca is a class library in python that can realize fast segmentation of fields. Exemplarily, the address data is "Haichuang Park, Yuhang District, Hangzhou City, Zhejiang Province". After segmentation by the segmentation tool cpca, the administrative district field of the address data is "Zhejiang Province", "Hangzhou City", "Yuhang District", and the specific address field is "Haichuang Park"; then the administrative district field can be spliced according to the splicing standard from large to small administrative district range, that is, in the order of provinces, cities and districts, and the segmentation result of the address data is output: the administrative district field is "Yuhang District, Hangzhou City, Zhejiang Province", and the specific address field is "Haichuang Park".
[0086] For example, Figure 3FIG. 2 shows a flow chart of address data segmentation according to an embodiment of the present invention. Figure 3 As shown, after obtaining a large amount of address data, data preprocessing is first performed, such as deduplication and data cleaning; then the address data is segmented through the CPCA module to determine the administrative region and specific address to which the address data belongs; then the fields of the administrative region are spliced according to the splicing standards of provinces, cities and districts, and finally the administrative region fields and specific address fields spliced according to provinces, cities and districts are output.
[0087] Step 1012: According to the administrative region field, the address data with the same administrative region field are divided into the same address partition.
[0088] In an embodiment of the present invention, after the address data is divided into an administrative region field and a specific address field, the address data of the same province, city or district can be divided into the same address partition through pyspark based on the administrative region field, and then the address data of the same province, city or district can be divided into the same task of spark for processing to realize distributed calculation of data.
[0089] In the embodiment of the present invention, the address data is segmented to obtain the administrative region field and the specific address field of the address data; based on the administrative region field, the address data with the same administrative region field are divided into the same address partition. In this way, the address data of the same province, city or district is divided into the same address partition, which can not only reduce the amount of data calculation, but also ensure that the real geographical locations of the address data with high similarity are close, thereby improving the accuracy of similar address screening.
[0090] Optionally, in the embodiment of the present invention, step 102 may specifically include the following steps 1021 to 1023:
[0091] Step 1021 , when there are high-frequency characters in the address data, extract the high-frequency characters in the address data as target address data; the high-frequency characters are address characters whose appearance frequency in all address data in the address partition is higher than a preset frequency threshold.
[0092] In the embodiment of the present invention, the preset frequency threshold may refer to a preset critical value of the frequency of occurrence of characters. The high-frequency characters may refer to characters whose frequency of occurrence in the address data of the address partition is higher than the preset frequency threshold. The target address data may refer to the address data used for similarity calculation.
[0093] In this step, since a massive amount of address data is obtained, the amount of address data in the same address partition may also be large. In order to improve the efficiency of data processing, it is necessary to extract high-frequency, well-distinguishing words to form a vocabulary. In this step, there may be multiple characters in an address data, and among the multiple characters, there may be characters with low frequency of occurrence and poor discrimination, such as rare characters, which can be discarded when extracting the target address data. Specifically, the high-frequency characters in the address data are extracted through regularization to form the target address data, so that the address data can be screened, the size of the address vector matrix finally generated can be limited, and the efficiency of subsequent data processing can be improved.
[0094] Step 1022: Add the target address data to the target word library.
[0095] In the embodiment of the present invention, the target word library may refer to a word library used to construct an address vector. The target word library may be formed based on the gensim.corpora.Dictionary method in natural language processing, and the embodiment of the present invention does not limit the specific method for generating the target word library.
[0096] Step 1023: Generate an address vector corresponding to each target address data in the target vocabulary by using a preset vector generation model to obtain the first address vector matrix.
[0097] In an embodiment of the present invention, the preset vector generation model may refer to a pre-set model for generating an address vector for each target address data in the target vocabulary. The preset vector generation model may refer to a term frequency-inverse document frequency model (TF-IDF), which can be implemented by gensim.models.TfidfModel in Pyspark. Of course, other vector generation methods may also be used to generate an address vector matrix, which is not limited in the embodiment of the present invention. The first address vector matrix may refer to a matrix composed of address vectors generated by each target address data in an address partition, and the first address vector matrix may be subsequently segmented and calculated.
[0098] In an embodiment of the present invention, when there are high-frequency characters in the address data, the high-frequency characters in the address data are extracted as target address data; the high-frequency characters are address characters whose frequency of appearance in all address data in the address partition is higher than a preset frequency threshold; the target address data is added to the target word library; and the address vector corresponding to each target address data in the target word library is generated through a preset vector generation model to obtain a first address vector matrix. In this way, in an embodiment of the present invention, the target address data is obtained by regular extraction of high-frequency characters in the address data, and the address vector is generated by the target address data in the target word library, which can reduce the amount of data calculation and improve data processing efficiency.
[0099] For example, Figure 4 FIG. 2 shows a schematic diagram of a process of generating a first address vector matrix according to an embodiment of the present invention. Figure 4 As shown, after obtaining the address data list in the address partition, an address dictionary can be formed based on the address data list. Afterwards, regular extraction is performed according to the rules of restricted words, which can limit the size of the final generated vector and avoid excessive calculations; regular extraction can also be performed according to the rules of unrestricted words, that is, extracting every character in each address data to ensure the comprehensiveness of data processing. Afterwards, based on the target address data obtained by regular extraction, a target vocabulary is formed, and then based on the target vocabulary, the bag of words model Dictionary.doc2bow in the gesim module in Pyspark is used to generate a vector corpus, and then the first address vector matrix is generated based on the preset vector generation model. It should be noted that Figure 4 The address word statistics shown in can be before regular extraction, that is, the frequency of occurrence of each address character in the address partition is first counted to determine the high-frequency characters, and then regular extraction is performed within the range of the high-frequency characters to determine the target address data; the address word statistics can also be after regular extraction, at this time, the regular extraction rules can be to limit the number of characters in a specific address field of the address data, etc. After regular extraction, the extracted address characters are counted again to determine the high-frequency characters and the target address data. Any method can be flexibly adopted based on the actual situation, and the embodiments of the present invention are not limited to this.
[0100] Optionally, in the embodiment of the present invention, step 104 may specifically include the following steps 1041 to 1045:
[0101] Step 1041 : Copy the first address vector matrix to obtain a second address vector matrix.
[0102] In the embodiment of the present invention, after obtaining the first address vector matrix, each row in the first address vector matrix is a target address data. In order to conveniently calculate the similarity between the target address data and the remaining target address data, the first address vector matrix can be copied to obtain a second address vector matrix.
[0103] Step 1042: divide the second address vector matrix into multiple second target vector matrices; the second target vector matrix has the same number of rows and columns as the first target vector matrix.
[0104] In an embodiment of the present invention, after copying the first address vector matrix to be calculated to obtain the second address vector matrix, the first address vector matrix and the second address vector matrix are cut according to the same batch size to obtain multiple first target vector matrices and multiple second target vector matrices.
[0105] In this step, when the matrix is cut, the number of rows and columns of the small matrix after cutting, i.e., the target vector matrix, can be controlled by controlling the batch_size parameter in the custom cutting function. In this way, the cutting of the matrix can be more accurate.
[0106] Step 1043: For any one of the first target vector matrix and any one of the second target vector matrix, determine the position of the calculation results of the first target vector matrix and the second target vector matrix in the result matrix.
[0107] In the embodiment of the present invention, the result matrix can be used to characterize the calculation result of the similarity between the first address vector matrix and the second address vector matrix. Since the first target vector matrix and the second target vector matrix are small matrices obtained after segmentation, the calculation result is a part of the result matrix directly calculated from the first address vector matrix and the second address vector matrix.
[0108] It should be noted that in order to improve calculation efficiency and avoid memory overflow, there is no need to actually calculate the result matrix in this step. It is only necessary to determine the positions of the calculation results of the first target vector matrix and the second target vector matrix in the result matrix based on the position of the first target vector matrix in the first address vector matrix and the position of the second target vector matrix in the second address vector matrix. This can be used to determine whether it is a repeated calculation later.
[0109] Specifically, in this step, since the first address vector matrix and the second address vector matrix are exactly the same, when calculating the similarity of any of the first target vector matrices obtained after segmentation and any of the second target vector matrices in pairs, there will be a repeated calculation process. Exemplarily, assuming that four first target vector matrices a, b, c, and d are obtained after the first address vector matrix is segmented, and four second target vector matrices A, B, C, and D are obtained after the second address vector matrix is segmented, then a and A, b and B, c and C, d and D are exactly the same in pairs. When calculating the similarity between the first target vector matrix a and the second target vector matrix B, the calculation result is repeated with the first target vector matrix b and the second target vector matrix A. Therefore, it is necessary to avoid such repeated calculations in the future to improve calculation efficiency.
[0110] Step 1044: If the calculation result is in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are input into a preset similarity model for calculation to obtain a result sub-matrix; the upper triangular position is a position in the result matrix where the number of rows is not greater than the number of columns.
[0111] In an embodiment of the present invention, the upper triangular position is a position in the upper triangular position in the result matrix, and in the upper triangular position, the number of rows is not greater than the number of columns. The preset similarity model may refer to a pre-set model for calculating the similarity between target address data. After the first target vector matrix and the second target vector matrix are input into the preset similarity model, a result sub-matrix may be obtained. Specifically, when calculating a small matrix, the distance calculation interface packaged in sklearn.metrics.pairwise may be used in Pyspark to calculate and obtain the final result sub-matrix, which is not limited in the embodiment of the present invention. The result sub-matrix may refer to a similarity matrix calculated from the first target vector matrix and the second target vector matrix, and each element in the result sub-matrix is an actual calculated value of the similarity.
[0112] Step 1045: If the calculation result is not in the upper triangle position of the result matrix, the first target vector matrix and the second target vector matrix are not calculated.
[0113] In the embodiment of the present invention, since the result matrix is a symmetric matrix with the main diagonal as the axis of symmetry and the corresponding elements are equal, only the upper triangular part needs to be calculated during the calculation. When the first target vector matrix and the second target vector matrix are not in the upper triangular position of the result matrix, it indicates that the calculation of the first target vector matrix and the second target vector matrix is a repeated calculation process, and the calculation may not be performed to improve the calculation efficiency.
[0114] In the embodiment of the present invention, the first address vector matrix is copied to obtain the second address vector matrix; the second address vector matrix is divided to obtain multiple second target vector matrices; for any first target vector matrix and any second target vector matrix, the position of the calculation results of the first target vector matrix and the second target vector matrix in the result matrix is determined; if the calculation result is in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are input into the preset similarity model for calculation to obtain a result sub-matrix; the upper triangular position is the position in the result matrix where the number of rows is not greater than the number of columns; if the calculation result is not in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are not calculated. In this way, in the embodiment of the present invention, by predicting the position of the first target vector matrix and the second target vector matrix in the result matrix, if they are in the upper triangular position, calculation is performed, and if they are not in the upper triangular position, no calculation is performed, thereby avoiding repeated calculation of symmetric matrices, preventing memory overflow problems caused by large calculation amounts, and improving calculation efficiency.
[0115] For example, Figure 5 FIG. 2 is a schematic diagram showing a similarity calculation process of a first address vector matrix according to an embodiment of the present invention. Figure 5 As shown, the first address vector matrix to be calculated is copied to obtain the second address vector matrix, and then the first address vector matrix is divided into four first target vector matrices a, b, c, and d, and the second address vector matrix is divided into four second target vector matrices A, B, C, and D. Afterwards, when any first target vector matrix and any second target vector matrix are calculated in pairs, the position of the calculation result in the result matrix is determined, and the calculation result is calculated when it is in the upper triangle of the result matrix, and the calculation result is not calculated when it is not in the upper triangle of the result matrix. In this way, repeated calculation of the upper and lower triangles of the symmetric matrix can be avoided, and the calculation efficiency can be improved.
[0116] Optionally, in the embodiment of the present invention, step 105 may specifically include the following steps 1051 to 1054:
[0117] Step 1051: for each result sub-matrix, determine whether each element in the result sub-matrix is greater than a preset similarity threshold; each element in the result sub-matrix is an address similarity.
[0118] In an embodiment of the present invention, the result submatrix may refer to a similarity result matrix output by a preset similarity model for characterizing target address data. Each element in the result submatrix characterizes the similarity between two target similar addresses. The preset similarity threshold may refer to a preset critical threshold for characterizing similarity, and the specific value of the threshold may be flexibly set based on actual conditions, which is not limited in the embodiment of the present invention. By comparing the specific value of the similarity corresponding to the element with the preset similarity threshold, it can be determined whether the two target address data corresponding to the element are similar.
[0119] Step 1052: When the element is greater than a preset similarity threshold, save the element and the row and column values of the similarity in the result sub-matrix.
[0120] In the embodiment of the present invention, the row and column values may refer to the number of rows and columns of an element, and based on the row and column values, two target address data in the first address vector matrix may be correspondingly determined as target similar address data. When an element in the result submatrix is higher than the preset similarity threshold, the two target address data corresponding to the element may be determined to be target similar addresses, and screening may be performed.
[0121] Step 1053: determine the target address data corresponding to the element in the first address vector matrix based on the row and column values, and save the target address data and the element to obtain target similar address data.
[0122] In an embodiment of the present invention, two target address data of the element in the first address vector matrix are determined based on the row and column values. At this time, the two target address data can be used as target similar address data, and the target similar address data and its similarity can be saved subsequently.
[0123] Step 1054: If the element is not greater than the preset similarity threshold, discard the element.
[0124] In the embodiment of the present invention, when the element is not greater than the preset similarity threshold, it indicates that the two target address data corresponding to the element have low similarity and can be directly discarded without being saved.
[0125] For example, refer to Figure 5 , for the result submatrix calculated by the first target vector matrix and the second target vector matrix, the computing device parses each element in the result submatrix, and when the similarity value of the element is greater than the preset similarity threshold, the element and the row and column values of the element are saved to filter out the target similar address data. In this way, there is no need to combine the result submatrices into a super large matrix for storage, saving system resources.
[0126] In an embodiment of the present invention, for each result submatrix, it is determined whether each element in the result submatrix is greater than a preset similarity threshold; each element in the result submatrix is an address similarity; when the element is greater than the preset similarity threshold, the row and column values of the element and the similarity in the result submatrix are saved; based on the row and column values, the target address data corresponding to the element is determined in the first address vector matrix, and the target address data and the element are saved to obtain the target similar address data; when the element is not greater than the preset similarity threshold, the element is discarded. In this way, in an embodiment of the present invention, by parsing the elements in each result submatrix and saving the target similar address data whose similarity is greater than the preset similarity threshold, there is no need to store the calculation results as a very large matrix, which can prevent the occurrence of insufficient memory and save system resources.
[0127] For example, Figure 6 FIG. 2 is a schematic diagram showing a processing flow of an address screening method according to an embodiment of the present invention. Figure 6 As shown, the computing device reads the massive address data through spark, and then performs preprocessing such as cleaning and deduplication on the massive address data; then, the massive address data is partitioned according to provinces, cities and districts through pyspark to obtain multiple address partitions; for each address partition, a first address vector matrix corresponding to each address partition is generated, and the first address vector matrix is cut and decomposed into smaller matrices, that is, multiple first target vector matrices (small matrix 1 to small matrix n). Then, the calculation results of the multiple first target vector matrices are calculated in the upper triangular position of the result matrix, and no calculation is performed when they are not in the upper triangular position; after the result sub-matrix is calculated, each element in the result sub-matrix is parsed to determine whether it is greater than the preset similarity threshold, and the result is retained and summarized in the case of being greater than, and the element is directly discarded in the case of being not greater than. In this way, the calculation of the similarity between massive addresses and the batch screening of similar addresses are realized, and the calculation efficiency is improved.
[0128] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.
[0129] Reference Figure 7 , shows a structural block diagram of a similar address screening device of the present invention. Specifically, the device 70 may include the following modules:
[0130] A partitioning module 701 is used to obtain a plurality of address data and partition the address data according to a preset rule to obtain a plurality of address partitions;
[0131] A generating module 702, configured to generate, for each address partition, a first address vector matrix corresponding to the address partition based on address data in the address partition;
[0132] A segmentation module 703 is used to segment the first address vector matrix to obtain a plurality of first target vector matrices;
[0133] A determination module 704, configured to determine the similarity between each address data in the address partition based on the multiple first target vector matrices;
[0134] The screening module 705 is used to screen out target similar address data based on the similarity.
[0135] In summary, the similar address screening device provided in the embodiment of the present invention obtains multiple address data, and partitions the multiple address data according to preset rules to obtain multiple address partitions; for each address partition, based on the address data in the address partition, generates a first address vector matrix corresponding to the address partition; divides the first address vector matrix to obtain multiple first target vector matrices; based on the multiple first target vector matrices, determines the similarity between each address data in the address partition; based on the similarity, screens out the target similar address data. In the embodiment of the present invention, by partitioning the address data to obtain multiple address partitions, and at the same time dividing and calculating the address vector matrix in each address partition, batch processing of a large number of addresses is realized, and addresses with high similarity can be screened out in batches, and because the calculation is performed on the divided small matrix, the resource occupancy is small, the calculation efficiency is high, and the operation cost is saved.
[0136] Optionally, the partition module 701 is specifically used to:
[0137] Segmenting the address data to obtain an administrative region field and a specific address field of the address data;
[0138] According to the administrative region field, the address data with the same administrative region field are divided into the same address partition.
[0139] Optionally, the generating module 702 is specifically configured to:
[0140] In the case where high-frequency characters exist in the address data, the high-frequency characters in the address data are extracted as target address data; the high-frequency characters are address characters whose frequency of appearance in all address data in the address partition is higher than a preset frequency threshold;
[0141] Adding the target address data to the target word library;
[0142] The address vector corresponding to each target address data in the target vocabulary is generated by a preset vector generation model to obtain the first address vector matrix.
[0143] Optionally, the determining module 704 is specifically configured to:
[0144] Copying the first address vector matrix to obtain a second address vector matrix;
[0145] Dividing the second address vector matrix to obtain a plurality of second target vector matrices; the second target vector matrix has the same number of rows and columns as the first target vector matrix;
[0146] For any one of the first target vector matrix and any one of the second target vector matrix, determine the position of the calculation result of the first target vector matrix and the second target vector matrix in the result matrix;
[0147] If the calculation result is in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are input into the preset similarity model for calculation to obtain a result sub-matrix; the upper triangular position is a position in the result matrix where the number of rows is not greater than the number of columns;
[0148] If the calculation result is not in the upper triangle position of the result matrix, the first target vector matrix and the second target vector matrix are not calculated.
[0149] Optionally, the screening module 705 is specifically used to:
[0150] For each result sub-matrix, determine whether each element in the result sub-matrix is greater than a preset similarity threshold; each element in the result sub-matrix is an address similarity;
[0151] When the element is greater than a preset similarity threshold, save the element and the row and column values of the similarity in the result submatrix;
[0152] Determine the target address data corresponding to the element in the first address vector matrix based on the row and column values, and save the target address data and the element to obtain target similar address data;
[0153] If the element is not greater than the preset similarity threshold, the element is discarded.
[0154] Optionally, an embodiment of the present invention further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned similar address screening method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0155] Optionally, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned similar address screening method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0156] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0157] It is easy for a person skilled in the art to think that any combination of the above embodiments is feasible, so any combination of the above embodiments is an implementation scheme of the present invention. However, due to space limitations, this specification will not describe them in detail here.
[0158] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.
[0159] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the intention that the claimed invention requires more features than those expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all of the features of the individual embodiments previously disclosed. Therefore, the claims that follow the detailed description are hereby expressly incorporated into the detailed description, with each claim itself serving as a separate embodiment of the present invention.
[0160] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.
[0161] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.
[0162] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.
Claims
1. A similar address screening method, It is characterized in that include: Acquire multiple address data, and partition the address data according to a preset rule to obtain multiple address partitions; For each address partition, based on the address data in the address partition, generate a first address vector matrix corresponding to the address partition; Dividing the first address vector matrix to obtain a plurality of first target vector matrices; Based on the multiple first target vector matrices, determining the similarity between each address data in the address partition; Based on the similarity, target similar address data is screened out; Partitioning the address data according to a preset rule includes: Segmenting the address data to obtain an administrative region field and a specific address field of the address data; According to the administrative region field, the address data with the same administrative region field are divided into the same address partition; The step of generating a first address vector matrix corresponding to the address partition based on the address data in the address partition comprises: In the case where high-frequency characters exist in the address data, extracting the high-frequency characters in the address data as target address data; the high-frequency characters are address characters whose frequency of appearance in the address partition is higher than a preset frequency threshold; Adding the target address data to the target word library; Generate an address vector corresponding to each target address data in the target word library by using a preset vector generation model to obtain the first address vector matrix; The determining the similarity between each address data in the address partition comprises: Copying the first address vector matrix to obtain a second address vector matrix; Dividing the second address vector matrix to obtain a plurality of second target vector matrices; the second target vector matrix has the same number of rows and columns as the first target vector matrix; For any one of the first target vector matrix and any one of the second target vector matrix, determine the position of the calculation result of the first target vector matrix and the second target vector matrix in the result matrix; If the calculation result is in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are input into the preset similarity model for calculation to obtain a result sub-matrix; the upper triangular position is a position in the result matrix where the number of rows is not greater than the number of columns; If the calculation result is not in the upper triangle position of the result matrix, the first target vector matrix and the second target vector matrix are not calculated.
2. The method according to claim 1, It is characterized in that The step of screening out target similar address data based on the similarity includes: For each result sub-matrix, determine whether each element in the result sub-matrix is greater than a preset similarity threshold; each element in the result sub-matrix is an address similarity; When the element is greater than a preset similarity threshold, save the element and the row and column values of the similarity in the result submatrix; Determine the target address data corresponding to the element in the first address vector matrix based on the row and column values, and save the target address data and the element to obtain target similar address data; If the element is not greater than the preset similarity threshold, the element is discarded.
3. A similar address screening device, It is characterized in that include: A partitioning module, used to obtain a plurality of address data, and partition the address data according to a preset rule to obtain a plurality of address partitions; A generating module, configured to generate, for each address partition, a first address vector matrix corresponding to the address partition based on address data in the address partition; A segmentation module, used for segmenting the first address vector matrix to obtain a plurality of first target vector matrices; A determination module, configured to determine the similarity between each address data in the address partition based on the multiple first target vector matrices; A screening module, used for screening out target similar address data based on the similarity; The partition module is specifically used for: Segmenting the address data to obtain an administrative region field and a specific address field of the address data; According to the administrative region field, the address data with the same administrative region field are divided into the same address partition; The generating module is specifically used for: In the case where high-frequency characters exist in the address data, the high-frequency characters in the address data are extracted as target address data; the high-frequency characters are address characters whose frequency of appearance in all address data in the address partition is higher than a preset frequency threshold; Adding the target address data to the target word library; Generate an address vector corresponding to each target address data in the target word library by using a preset vector generation model to obtain the first address vector matrix; The device is also used for: Copying the first address vector matrix to obtain a second address vector matrix; Dividing the second address vector matrix to obtain a plurality of second target vector matrices; the second target vector matrix has the same number of rows and columns as the first target vector matrix; For any one of the first target vector matrix and any one of the second target vector matrix, determine the position of the calculation result of the first target vector matrix and the second target vector matrix in the result matrix; If the calculation result is in the upper triangular position of the result matrix, the first target vector matrix and the second target vector matrix are input into the preset similarity model for calculation to obtain a result sub-matrix; the upper triangular position is a position in the result matrix where the number of rows is not greater than the number of columns; If the calculation result is not in the upper triangle position of the result matrix, the first target vector matrix and the second target vector matrix are not calculated.
4. A computing device, It is characterized in that The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program implements the similar address screening method according to any one of claims 1 to 2 when executed by the processor.
5. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the similar address screening method according to any one of claims 1 to 2 is implemented.
