Repeated data judgment method and device for business opportunity data, equipment and medium

Abstract

The invention relates to a repeated data judgment method and device for business opportunity data, equipment and a medium. After the business opportunity data is obtained, multiple pieces of business opportunity historical data meeting screening conditions are selected from a business opportunity historical data set according to the screening conditions; the business opportunity data and text data and non-text data in the multiple pieces of business opportunity historical data are processed respectively to obtain similarity features and specified features, then the similarity between the business opportunity data and each piece of business opportunity historical data is calculated according to the similarity features and the specified features by adopting a weight-optimized Euclidean distance formula, and the similarity between the business opportunity data and each piece of business opportunity historical data is calculated according to the similarity features and the specified features. According to the similarity, whether the business opportunity data is suspected repeated data is judged through a preset threshold value, and if the business opportunity data is judged to be suspected repeated data, a repetition warning is sent. By adopting the method, accurate duplicate judgment can be performed on business opportunity data, and meanwhile, the implementation of the method does not need to occupy a large amount of storage space of hardware.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and medium for determining duplicate data of business opportunity data. Background Art

[0002] With the development of economy and the acceleration of globalization, market competition is becoming more and more fierce. Many enterprises and individuals are looking for new business opportunities to gain competitive advantages. Within the group, judging whether a business opportunity already exists or has been developed and utilized by others can help group enterprises or individuals avoid duplication of work and waste of resources.

[0003] At present, the repeated judgment of business opportunity management for enterprises is more often done by direct database SQL query processing, or even more so by human judgment, which has problems such as inaccurate judgment of duplicate data and low efficiency. In particular, group enterprises are prone to missed judgment of repeated business opportunities due to information gaps between them and their subsidiaries at all levels. In serious cases, they may be directly eliminated from the competition due to being identified as collusion in bidding for the same project. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, equipment and medium for determining duplicate data of business opportunity data with improved accuracy in response to the above technical problems.

[0005] A method for determining duplicate data in business opportunity data, the method comprising:

[0006] Acquire business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information;

[0007] Acquire a screening condition, and select multiple pieces of business opportunity historical data that meet the screening condition from the business opportunity historical data set according to the screening condition;

[0008] Processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using the similarity as a similarity feature;

[0009] Acquire a designated feature type of a non-text type, and perform normalization processing on the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type to obtain designated features corresponding to the business opportunity data and the plurality of business opportunity historical data;

[0010] The weighted optimized Euclidean distance formula is used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature value;

[0011] According to the similarity, whether the business opportunity data is suspected duplicate data is determined by a preset threshold, and if it is determined to be suspected duplicate data, a duplicate warning is sent.

[0012] In one embodiment, the business opportunity information includes: business opportunity source, business opportunity name, business opportunity date;

[0013] The customer information includes: customer name, customer address, customer source and contact information.

[0014] In one embodiment, the similarity feature is obtained by processing the business opportunity data and the text data in the plurality of business opportunity history data, including:

[0015] Performing word segmentation processing on the business opportunity data and the text data in the multiple business opportunity historical data, and synthesizing a corpus according to the word segmentation processing results, wherein the corpus includes multiple rows of data corresponding to the word segmentation results of the business opportunity data and the multiple business opportunity historical data respectively;

[0016] Based on the corpus, the TF-IDF value of each word in each row of data is calculated, and each TF-IDF value is used as a weight value to construct a weight matrix;

[0017] Based on the weight matrix, the similarity between the business opportunity data and each piece of the business opportunity history data is calculated using cosine similarity, and is used as the similarity feature.

[0018] In one embodiment, the calculating the similarity between the business opportunity data and each piece of the business opportunity history data using cosine similarity based on the weight matrix includes:

[0019] In the weight matrix, a row of weight data corresponding to the business opportunity data and each piece of the business opportunity historical data is used to construct a weight data set;

[0020] The cosine similarity is calculated based on the weighted data set corresponding to the business opportunity data and the weighted data set corresponding to each piece of the business opportunity historical data.

[0021] In one embodiment, the cosine similarity between the business opportunity data and the business opportunity historical data is calculated using the following formula:

[0022]

[0023] In the above formula, A and B represent weighted data sets of the business opportunity data and the business opportunity history data, respectively, and n represents the number of words in the corpus.

[0024] In one embodiment, the non-text type of specified feature types includes opportunity amount, opportunity date, and region;

[0025] The specified feature types include one or more types.

[0026] In one embodiment, the weight-optimized Euclidean distance formula is expressed as:

[0027]

[0028] In the above formula, X and Y represent the characteristic values ​​of the business opportunity data and the business opportunity historical data respectively, w represents the weight, W represents the normalization factor, P represents the adjustment weight, the subscript i represents a certain characteristic value, and n represents the number of characteristic value types.

[0029] The present application also provides a device for determining duplicate data of business opportunity data, the device comprising:

[0030] A module for acquiring data to be judged, used for acquiring business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information;

[0031] A historical data screening module, used to obtain screening conditions, and select multiple business opportunity historical data that meet the screening conditions from the business opportunity historical data set according to the screening conditions;

[0032] A similarity feature obtaining module, used for processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using it as the similarity feature;

[0033] A designated feature obtaining module, used for obtaining a designated feature type of a non-text type, normalizing the business opportunity data and corresponding data in a plurality of business opportunity historical data according to the designated feature type, and obtaining designated features corresponding to the business opportunity data and the plurality of business opportunity historical data;

[0034] A similarity obtaining module, used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature by using the weighted optimized Euclidean distance formula;

[0035] The duplicate data judgment module is used to judge whether the business opportunity data is suspected duplicate data according to the similarity through a preset threshold, and send a duplicate warning if it is judged to be suspected duplicate data.

[0036] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0037] Acquire business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information;

[0038] Acquire a screening condition, and select multiple pieces of business opportunity historical data that meet the screening condition from the business opportunity historical data set according to the screening condition;

[0039] Processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using the similarity as a similarity feature;

[0040] Acquire a designated feature type of a non-text type, and perform normalization processing on the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type to obtain designated features corresponding to the business opportunity data and the plurality of business opportunity historical data;

[0041] The weighted optimized Euclidean distance formula is used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature value;

[0042] According to the similarity, whether the business opportunity data is suspected duplicate data is determined by a preset threshold, and if it is determined to be suspected duplicate data, a duplicate warning is sent.

[0043] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0044] Acquire business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information;

[0045] Acquire a screening condition, and select multiple pieces of business opportunity historical data that meet the screening condition from the business opportunity historical data set according to the screening condition;

[0046] Processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using the similarity as a similarity feature;

[0047] Acquire a designated feature type of a non-text type, and perform normalization processing on the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type to obtain designated features corresponding to the business opportunity data and the plurality of business opportunity historical data;

[0048] The weighted optimized Euclidean distance formula is used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature value;

[0049] According to the similarity, whether the business opportunity data is suspected duplicate data is determined by a preset threshold, and if it is determined to be suspected duplicate data, a duplicate warning is sent.

[0050] The above-mentioned duplicate data judgment method, device, equipment and medium for business opportunity data, after obtaining the business opportunity data, selects multiple business opportunity historical data that meet the screening conditions from the business opportunity historical data set according to the screening conditions, and processes the business opportunity data and the text data and non-text data in the multiple business opportunity historical data separately to obtain similarity features and specified features, and then uses the weighted optimized Euclidean distance formula to calculate the similarity between the business opportunity data and each business opportunity historical data according to the similarity feature and the specified feature value, and judges whether the business opportunity data is suspected duplicate data based on the similarity through a preset threshold, and sends a duplicate warning if it is suspected duplicate data. This method can be used to accurately judge the duplicate business opportunity data, and at the same time, the implementation of this method does not require a large amount of hardware storage space. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flowchart of a method for determining duplicate data of business opportunity data in one embodiment;

[0052] Figure 2 A flowchart of a method for determining duplicate data of business opportunity data in another embodiment;

[0053] Figure 3 It is a structural block diagram of a device for determining duplicate data of business opportunity data in one embodiment;

[0054] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] In the modern business environment, business opportunity data is an important part of business operations and marketing. Business opportunity data includes contact information, needs and behaviors of potential customers, which is crucial to the development and growth of enterprises. However, the existence of business opportunity data duplication seriously affects the operational efficiency and market competitiveness of enterprises.

[0057] First, the duplication of business opportunity data leads to a waste of corporate resources. When business opportunity data is duplicated, companies tend to repeatedly invest resources to handle the same business opportunities, including marketing activities, sales team time and energy, and marketing expenses. This not only wastes corporate resources, but also reduces corporate efficiency and productivity, affecting corporate profitability.

[0058] Secondly, the duplication of business opportunity data affects the company's ability to understand customers and provide personalized services. The duplication of business opportunity data makes it impossible for companies to accurately understand customer needs and preferences and provide personalized products and services. This will reduce customer satisfaction and affect customer loyalty and willingness to cooperate. In a highly competitive market environment, personalized services are an important means for companies to gain competitive advantages and retain customers. The duplication of business opportunity data seriously affects the market competitiveness of companies.

[0059] In addition, the duplication of business opportunity data can also damage the company's reputation. When a group company provides duplicate services to customers, customers may feel confused and dissatisfied. This will not only reduce customers' trust and goodwill in the company, but also have a negative impact on the company's reputation.

[0060] Finally, the duplication of business opportunity data will also have a negative impact on the company's decision-making and strategic planning. Business opportunity data is an important basis for companies to formulate marketing strategies and business development plans. However, when business opportunity data is duplicated, it is difficult for corporate decision makers to accurately analyze and understand market needs and trends. This will cause companies to make wrong decisions, miss market opportunities, and affect the long-term development of the company.

[0061] In summary, the seriousness of the problem of business opportunity data duplication cannot be ignored. It not only wastes the company's resources, reduces the company's operational efficiency and market competitiveness, but also affects the company's ability to understand customers and provide personalized services, damages the company's reputation, and has a negative impact on the company's decision-making and strategic planning. Therefore, companies should pay attention to the management and cleaning of business opportunity data, ensure the accuracy and integrity of the data, and improve the company's operational efficiency and market competitiveness.

[0062] However, when using existing data duplication detection technology in the duplication detection scenario of business opportunity data, there are some defects, which limits its application in large-scale data processing and privacy protection. The following is a detailed description of the defects of existing data duplication detection technology:

[0063] First, existing data duplication detection technologies often have high algorithmic complexity. Most duplication detection algorithms are based on hash algorithms, string matching algorithms, or feature extraction algorithms. These algorithms require a long computing time when processing large-scale data sets, resulting in high algorithmic complexity and failure to meet real-time requirements. For scenarios that require fast duplication detection, the efficiency of existing technologies is not high enough.

[0064] Secondly, existing data duplication detection technologies require a large amount of storage space. Duplicate detection algorithms usually need to store duplicated data information in memory or disk for quick search. For large-scale data sets, a large amount of storage resources is consumed, increasing storage costs. In addition, as the amount of data increases, the storage space requirements will also increase exponentially, limiting the scalability of duplicate detection technology.

[0065] Third, existing data duplication detection technologies have certain limitations in terms of accuracy. For data with high similarity, existing algorithms often cannot accurately determine whether it is duplicate data, resulting in low accuracy in duplication detection. For example, in text data, when only part of the content is the same or there is a high degree of similarity, existing algorithms often cannot accurately determine whether it is duplicate data. This is a major limitation for scenarios that require high-precision duplication detection, such as copyright protection and text deduplication.

[0066] Fourth, existing data duplication detection technologies have high requirements for the amount of data. For small-scale data sets, the existing technology has a good duplication detection effect, but for large-scale data sets, the duplication detection effect is poor. This is because for large-scale data sets, the algorithm complexity is high, and the time and space overhead of duplication detection is large. This limits the application of duplication detection technology in big data environments.

[0067] Fifth, existing data duplication detection technologies have high requirements for data updates. For newly added data or data sets with large changes, duplicate detection needs to be performed again, resulting in low duplicate detection efficiency. This is a major challenge for scenarios that require real-time updates, such as streaming data deduplication and online data deduplication.

[0068] In summary, the existing data duplication detection technology has defects such as high algorithm complexity, large storage space occupation, low duplication detection accuracy, high requirements for data format, high requirements for data volume, and high requirements for data update. It is not suitable for the application scenario of business opportunity data duplication detection. In order to solve these problems, Figure 1 As shown, in this embodiment, a method for determining duplicate data for business opportunity data is proposed, which specifically includes the following steps:

[0069] Step S100, obtaining business opportunity data, where the business opportunity data includes business opportunity information and corresponding customer information.

[0070] Step S110, obtaining a screening condition, and selecting a plurality of business opportunity historical data that meet the screening condition from the business opportunity historical data set according to the screening condition.

[0071] Step S120 , processing the business opportunity data and the text data in the plurality of business opportunity history data to obtain the similarity between the business opportunity data and each piece of business opportunity history data, and using it as a similarity feature.

[0072] Step S130, obtaining a designated feature type of a non-text type, normalizing the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type, and obtaining designated features corresponding to the business opportunity data and the plurality of business opportunity historical data.

[0073] Step S140, using the weighted optimized Euclidean distance formula according to the similarity feature and the specified feature value, calculate the similarity between the business opportunity data and each piece of business opportunity history data.

[0074] Step S150: judging whether the business opportunity data is suspected duplicate data according to the similarity through a preset threshold, and sending a duplicate warning if it is suspected duplicate data.

[0075] In this application, in response to the above-mentioned technical defects, a set of simple and efficient data duplication detection technologies is provided to simplify the development process and lower the technical threshold; at the same time as the data entry is completed, data duplication detection is performed in real time to achieve efficient data duplication detection.

[0076] In step S100, new business opportunity data is entered through a computer or tablet equipped with a system related to business opportunity data. The business opportunity data includes business opportunity information and customer information related to the business opportunity information, wherein the business opportunity information includes: business opportunity source, business opportunity name, business opportunity date. Customer information includes: customer name, customer address, customer source and contact information.

[0077] Since historical business opportunity data is usually large-scale data, when judging whether the input data is repeated with the historical business opportunity data, you can first perform a first screening in the historical business opportunity data according to the input screening conditions, screen out a part of the historical business opportunity data, and then judge the similarity between this part of the historical business opportunity data and the input data.

[0078] In step S110, the screening condition may be a time range screening, or a customer information or contact screening. According to the screening condition, a plurality of business opportunity historical data meeting the condition are screened out from the historical business opportunity data, thereby obtaining a sample business opportunity historical data set.

[0079] Specifically, the process of filtering historical business opportunity data according to the filtering conditions can be expressed as follows:

[0080]

[0081] In the above formula, T i represents the business opportunity historical data, n represents the name of each business opportunity historical data set in the business opportunity historical data, S j Indicates the filtering condition, and n1 indicates the condition name, such as "year>=2022".

[0082] In this embodiment, considering that the business opportunity data includes different types of data, generally including text data and non-text data, wherein the non-text data refers to numerical data. Considering that if only the text data type is judged, the judgment result will not be very accurate, in this embodiment, the similarity of the text data is first judged, and the judgment result is used as the feature value, and then the numerical data is processed, and the final similarity judgment is performed in combination with the judgment result of the text data to improve the accuracy of the judgment result.

[0083] In step S120, the text data in the business opportunity data and the multiple business opportunity historical data are processed to obtain similarity features, including: first, the business opportunity data and the text data in the multiple business opportunity historical data are segmented, and a corpus is synthesized according to the word segmentation results, the corpus includes multiple rows of data corresponding to the word segmentation results of the business opportunity data and the multiple business opportunity historical data, then, based on the corpus, the TF-IDF value of each word in each row of data is calculated, and each TF-IDF value is used as a weight value to construct a weight matrix, and then based on the weight matrix, the cosine similarity is used to calculate the similarity between the business opportunity data and each business opportunity historical data, and it is used as the similarity feature.

[0084] In this embodiment, the text data in the business opportunity data and the multiple business opportunity historical data obtained after screening are segmented using a Chinese word segmentation method based on a hidden Markov model. The basic idea is to regard the text data to be segmented as an observation sequence, and the Chinese words as a hidden state sequence. By statistically learning the observation sequence, the most likely state sequence (i.e., word sequence) is inferred, thereby achieving Chinese word segmentation) and obtaining the segmentation results of the entire text. The multiple words obtained after the business opportunity data and the multiple business opportunity historical data are segmented are merged into a corpus.

[0085] Specifically, the obtained corpus includes multiple lines of data, each line corresponding to a word segmentation result of a piece of business opportunity data or business opportunity history data.

[0086] Furthermore, the frequency of occurrence of each word in each row of data is calculated based on the corpus, that is, the frequency of occurrence of each word in a piece of business opportunity data or business opportunity history data, that is, the TF value, and the calculation adopts the following formula:

[0087] Right now

[0088] Next, the rarity of a word in the entire corpus is calculated based on the corpus, that is, the inverse document frequency, IDF value. The calculation formula of IDF is the total number of documents in the corpus divided by the logarithm of the number of documents containing the word, where the document refers to the business opportunity data or a piece of business opportunity history data. If there are fewer documents containing word t, the larger the IDF, it means that the entry has a good ability to distinguish categories.

[0089] Specifically, the following formula is used to calculate IDF:

[0090] Right now

[0091] Furthermore, after obtaining the TF value and IDF value of each word in each business opportunity data and each business opportunity history data, the product of the TF value and the IDF value, namely the TF-IDF value, is calculated to measure the importance of a word in a document. A high TF-IDF value indicates the importance of the word to the document, i.e., a certain business opportunity data, while a low value indicates low importance.

[0092] In this embodiment, a weight matrix is ​​constructed using the TF-IDF value of each word in the business opportunity data and each piece of business opportunity history data. Each row of this matrix represents a corresponding piece of business opportunity data or business opportunity history data, and the column represents the vocabulary obtained by word segmentation, and each element in the matrix represents the TF-IDF value of the word in the corresponding data.

[0093] Furthermore, based on the weight matrix, the cosine similarity is used to calculate the similarity between the business opportunity data and each piece of business opportunity historical data, including: in the weight matrix, a weight data set is constructed by respectively corresponding a row of weight data corresponding to the business opportunity data and each piece of business opportunity historical data, and the cosine similarity is calculated based on the weight data set corresponding to the business opportunity data and the weight data set corresponding to each piece of business opportunity historical data.

[0094] Specifically, the cosine similarity between business opportunity data and business opportunity historical data is calculated using the following formula:

[0095]

[0096] In the above formula, A and B represent the weighted data sets of business opportunity data and a piece of business opportunity history data respectively, and n represents the number of words in the corpus.

[0097] Through the above calculation, the similarity between the input business opportunity data and multiple business opportunity historical data calculated through text data can be obtained. However, it is inaccurate to judge the duplicate data only based on the similarity between text data. Therefore, in this method, the similarity obtained based on the text data is used as the similarity feature for subsequent judgment.

[0098] Considering that the business opportunity data includes non-text data in addition to text data, the present method also performs similarity judgment based on the non-text data.

[0099] Specifically, non-text data includes but is not limited to opportunity amount, opportunity date, and region. The specified feature type may include one or more types, that is, in the subsequent similarity calculation, the numerical data participating in the similarity calculation may be one or more types.

[0100] In this embodiment, relevant data is selected from the business opportunity data and multiple business opportunity history data according to the acquired specified feature type to calculate the similarity, and the process is expressed as follows:

[0101]

[0102] In the above formula, S i It represents the selected feature set (such as opportunity amount, similarity of opportunity information, etc.). In the actual calculation process, some feature values ​​may not be selected well, which may lead to inaccurate algorithm. n represents the specified feature type obtained.

[0103] Since the numerical data includes data in different units such as amounts and dates, which have a great impact on the subsequent distance calculation, in this embodiment, the selected feature values ​​need to be normalized. The main normalization formula is expressed as:

[0104] y3=(XX min ) / (X max -X min )

[0105] In the above formula, X represents the data to be normalized, X min is the minimum value of the selected feature column, X max is the maximum value of the selected feature column.

[0106] Next, the normalized numerical data is used as the designated feature. When there are multiple types of designated features, multiple designated features corresponding to different categories are obtained.

[0107] In step 140, based on the similarity features obtained in the previous steps and the specified features, the distance between the business opportunity data and each piece of business opportunity history data, ie, the final similarity, is calculated using the Euclidean distance calculation method.

[0108] Euclidean distance is the straight-line distance between two points. The traditional Euclidean distance calculation formula is:

[0109]

[0110] However, this formula is not accurate enough for actual data processing. In real life, if two identical features are closer, their influence is greater, so the weight value should be larger. For example, if there are three companies, the corresponding attribute values ​​are company A: [3,5,8], B: [1,-1,3], C [3,4,0], and the Euclidean distance is calculated as the distance between A and B is the same as the distance between A and C. But in fact, the similarity between A and C should be greater, because two of the values ​​are very similar, so the Euclidean distance algorithm is optimized, mainly by adding weights to make the data and single feature values ​​more similar. After weighting, the result of A and B is 2.35, and the distance between A and C is 0.53. Therefore, it is concluded that the distance between A and C is closer. The optimized Euclidean distance calculation formula is as follows:

[0111]

[0112] In the above formula, X and Y represent the characteristic values ​​of the business opportunity data and the business opportunity historical data respectively, w represents the weight, W represents the normalization factor, P represents the adjustment weight, the subscript i represents a certain characteristic value, and n represents the number of characteristic value types.

[0113] It should be noted here that when i represents a similarity feature, x takes the value of 1 and y takes the value of the corresponding similarity feature.

[0114] Furthermore, w, W, and P are adjusted according to different characteristics, where w i According to the calculation formula We get: where σ represents the adjustment factor and W is calculated according to the formula When calculating these parameters, they can be adjusted continuously based on experience and the algorithm's fitting process to achieve the optimum.

[0115] The distance between the business opportunity data and each piece of business opportunity historical data is calculated by the above-mentioned optimized Euclidean distance calculation formula, and the obtained distances are arranged in ascending order. In step S150, if the minimum distance value is less than the preset threshold, it means that the currently entered business opportunity data is likely to be significantly different from the business opportunity historical data corresponding to the minimum distance value. In this case, a duplicate warning needs to be sent to remind that the currently entered data is duplicate data, and a second manual judgment is performed, thereby realizing a business opportunity duplicate warning.

[0116] like Figure 2 The figure shows a flowchart of a method for determining duplicate data for business opportunity data.

[0117] In the above-mentioned duplicate data judgment method for business opportunity data, according to the characteristics of business opportunity data, it is intended to select multiple business opportunity historical data that meet the filtering conditions from the business opportunity historical data set according to the filtering conditions, and process the business opportunity data and the text data and non-text data in the multiple business opportunity historical data separately to obtain similarity features and specified features, and then use the weighted optimized Euclidean distance formula to calculate the similarity between the business opportunity data and each business opportunity historical data according to the similarity features and the specified feature values, and judge whether the business opportunity data is suspected duplicate data based on the similarity through the preset threshold, and send a duplicate warning if it is suspected duplicate data. This method can be used to accurately judge the duplicate data of business opportunity data, and at the same time, the implementation of this method does not require a large amount of hardware storage space. This method can effectively solve the defects of the current deduplication data solution when it is applied to business opportunity data, such as high algorithm complexity, large storage space occupation, low duplication judgment accuracy, high requirements for data format, high requirements for data volume, and high requirements for data update.

[0118] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0119] In one embodiment, Figure 3 As shown, a duplicate data judgment device for business opportunity data is provided, comprising: a to-be-judged data acquisition module 200, a historical data screening module 210, a similarity feature acquisition module 220, a specified feature acquisition module 230, a similarity acquisition module 240 and a duplicate data judgment module 250, wherein:

[0120] The to-be-judged data acquisition module 200 is used to acquire business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information;

[0121] The historical data screening module 210 is used to obtain screening conditions and select multiple business opportunity historical data that meet the screening conditions from the business opportunity historical data set according to the screening conditions;

[0122] A similarity feature obtaining module 220 is used to process the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and use it as a similarity feature;

[0123] The designated feature obtaining module 230 is used to obtain a designated feature type of a non-text type, and normalize the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type to obtain designated features corresponding to the business opportunity data and the plurality of business opportunity historical data;

[0124] A similarity obtaining module 240 is used to calculate the similarity between the business opportunity data and each piece of the business opportunity history data according to the similarity feature and the specified feature by using the weight-optimized Euclidean distance formula;

[0125] The duplicate data judging module 250 is used to judge whether the business opportunity data is suspected duplicate data according to the similarity through a preset threshold, and send a duplicate warning if it is determined to be suspected duplicate data.

[0126] For the specific limitations of the duplicate data judgment device for business opportunity data, please refer to the limitations of the duplicate data judgment method for business opportunity data above, which will not be repeated here. Each module in the above-mentioned duplicate data judgment device for business opportunity data can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0127] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for determining duplicate data for business opportunity data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0128] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0130] Acquire business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information;

[0131] Acquire a screening condition, and select multiple pieces of business opportunity historical data that meet the screening condition from the business opportunity historical data set according to the screening condition;

[0132] Processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using the similarity as a similarity feature;

[0133] Acquire a designated feature type of a non-text type, and perform normalization processing on the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type to obtain designated features corresponding to the business opportunity data and the plurality of business opportunity historical data;

[0134] The weighted optimized Euclidean distance formula is used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature value;

[0135] According to the similarity, whether the business opportunity data is suspected duplicate data is determined by a preset threshold, and if it is determined to be suspected duplicate data, a duplicate warning is sent.

[0136] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0137] Acquire business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information;

[0138] Acquire a screening condition, and select multiple pieces of business opportunity historical data that meet the screening condition from the business opportunity historical data set according to the screening condition;

[0139] Processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using the similarity as a similarity feature;

[0140] Acquire a designated feature type of a non-text type, and perform normalization processing on the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type to obtain designated features corresponding to the business opportunity data and the plurality of business opportunity historical data;

[0141] The weighted optimized Euclidean distance formula is used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature value;

[0142] According to the similarity, whether the business opportunity data is suspected duplicate data is determined by a preset threshold, and if it is determined to be suspected duplicate data, a duplicate warning is sent.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0144] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for determining duplicate data of business opportunity data, characterized in that: The method comprises: Acquire business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information; Acquire a screening condition, and select multiple pieces of business opportunity historical data that meet the screening condition from the business opportunity historical data set according to the screening condition; Processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using the similarity as a similarity feature; Acquire a designated feature type of a non-text type, and perform normalization processing on the business opportunity data and corresponding data in the plurality of business opportunity historical data according to the designated feature type to obtain designated features corresponding to the business opportunity data and the plurality of business opportunity historical data; The weighted optimized Euclidean distance formula is used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature value; According to the similarity, whether the business opportunity data is suspected duplicate data is determined by a preset threshold, and if it is determined to be suspected duplicate data, a duplicate warning is sent.

2. The method for determining duplicate data of business opportunity data according to claim 1, characterized in that: The business opportunity information includes: business opportunity source, business opportunity name, and business opportunity date; The customer information includes: customer name, customer address, customer source and contact information.

3. The method for determining duplicate data of business opportunity data according to claim 2, characterized in that: The similarity feature is obtained by processing the business opportunity data and the text data in the plurality of business opportunity history data, including: Performing word segmentation processing on the business opportunity data and the text data in the multiple business opportunity historical data, and synthesizing a corpus according to the word segmentation processing results, wherein the corpus includes multiple rows of data corresponding to the word segmentation results of the business opportunity data and the multiple business opportunity historical data respectively; Based on the corpus, the TF-IDF value of each word in each row of data is calculated, and each TF-IDF value is used as a weight value to construct a weight matrix; Based on the weight matrix, the similarity between the business opportunity data and each piece of the business opportunity history data is calculated using cosine similarity, and is used as the similarity feature.

4. The method for determining duplicate data of business opportunity data according to claim 3, characterized in that: The calculating the similarity between the business opportunity data and each piece of the business opportunity history data by using cosine similarity based on the weight matrix includes: In the weight matrix, a row of weight data corresponding to the business opportunity data and each piece of the business opportunity historical data is used to construct a weight data set; The cosine similarity is calculated based on the weighted data set corresponding to the business opportunity data and the weighted data set corresponding to each piece of the business opportunity historical data.

5. The method for determining duplicate data of business opportunity data according to claim 4, characterized in that: The cosine similarity between the business opportunity data and the business opportunity historical data is calculated using the following formula: In the above formula, A and B represent weighted data sets of the business opportunity data and the business opportunity history data, respectively, and n represents the number of words in the corpus.

6. The method for determining duplicate data of business opportunity data according to claim 1, characterized in that: The specified feature types of the non-text type include opportunity amount, opportunity date, and region; The specified feature types include one or more types.

7. The method for determining duplicate data of business opportunity data according to claim 1, characterized in that: The Euclidean distance formula after weight optimization is expressed as: In the above formula, X and Y represent the characteristic values ​​of the business opportunity data and the business opportunity historical data respectively, w represents the weight, W represents the normalization factor, P represents the adjustment weight, the subscript i represents a certain characteristic value, and n represents the number of characteristic value types.

8. A device for determining duplicate data of business opportunity data, characterized in that: The device comprises: A module for acquiring data to be judged, used for acquiring business opportunity data, wherein the business opportunity data includes business opportunity information and corresponding customer information; A historical data screening module, used to obtain screening conditions, and select multiple business opportunity historical data that meet the screening conditions from the business opportunity historical data set according to the screening conditions; A similarity feature obtaining module, used for processing the business opportunity data and the text data in the plurality of business opportunity historical data to obtain the similarity between the business opportunity data and each of the business opportunity historical data, and using it as the similarity feature; A designated feature obtaining module, used for obtaining a designated feature type of a non-text type, normalizing the business opportunity data and corresponding data in a plurality of business opportunity historical data according to the designated feature type, and obtaining designated features corresponding to the business opportunity data and the plurality of business opportunity historical data; A similarity obtaining module, used to calculate the similarity between the business opportunity data and each piece of the business opportunity historical data according to the similarity feature and the specified feature by using the weighted optimized Euclidean distance formula; The duplicate data judgment module is used to judge whether the business opportunity data is suspected duplicate data according to the similarity through a preset threshold, and send a duplicate warning if it is judged to be suspected duplicate data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images