Big data-based data cleaning method and system

Abstract

The present application relates to the technical field of data cleaning, and particularly relates to a data cleaning method and system based on big data, which comprises the following steps: determining a user platform use index corresponding to each target user according to the platform use condition of each target user who places an order during the official period of a to-be-monitored activity in the preheating period of the to-be-monitored activity; determining a user platform active index and a user behavior depth index corresponding to each target user, thereby determining a possible order brushing index corresponding to each target user; screening out real order brushing users; removing the to-be-monitored activity participation information corresponding to all real order brushing users from the to-be-monitored activity participation information corresponding to all target users, thereby realizing data cleaning. The present application realizes the identification of real order brushing users relatively accurately by analyzing the platform use condition, commodity browsing condition, commodity consultation and collection condition and order distribution condition during the activity period, thereby improving the accuracy of data cleaning.

Description

Technical Field

[0001] This invention relates to the field of data cleaning technology, and more specifically to data cleaning methods and systems based on big data. Background Technology

[0002] In the process of generating big data, there is often abnormal data that needs to be cleaned out. For example, in the large amount of data generated during e-commerce events, there is often data related to abnormal behaviors such as fraudulent orders. To facilitate subsequent analysis of genuine user feedback generated during e-commerce events, it is often necessary to remove the data generated by these abnormal behaviors, thus achieving data cleansing. Currently, the common method for data cleansing is to use anomaly detection algorithms to identify and delete abnormal data. Among these, the Isolation Forest algorithm is a commonly used anomaly detection algorithm.

[0003] However, when using anomaly detection algorithms to identify fraudulent order data within the large volume of e-commerce activity-related data generated during major e-commerce events, the following technical problems often arise:

[0004] Most existing anomaly detection algorithms classify few but distinct data points as anomalous. For example, the Isolation Forest algorithm often identifies few but distinct data points as anomalous. However, during large-scale e-commerce events, a large number of fraudulent orders often occur on e-commerce platforms. At this time, such anomalous behavior is no longer characterized by few but distinct data points. If the Isolation Forest algorithm is used to clean the fraudulent order data, it may be difficult to accurately identify the fraudulent order data, resulting in poor accuracy of data cleaning. Summary of the Invention

[0005] To address the technical problem of poor data cleaning accuracy caused by poor accuracy in identifying fraudulent order data, this invention proposes a data cleaning method and system based on big data.

[0006] In a first aspect, the present invention provides a data cleaning method based on big data, the method comprising:

[0007] Based on the platform usage of each target user who placed an order during the official period of the monitored activity, and the platform usage during the pre-heating period of the monitored activity, determine the corresponding user platform usage indicators for each target user.

[0008] Based on the user platform usage metrics corresponding to each target user, and the browsing data of the same category of products of each target user during the pre-heating period of the activity to be monitored, determine the user platform activity metrics corresponding to each target user.

[0009] Based on the product inquiries and collections of each target user during the pre-heating period of the activity to be monitored, and the user platform activity indicators corresponding to each target user, the user behavior depth indicators corresponding to each target user are determined.

[0010] Based on the order distribution of each target user during the official period of the monitored activity, the user behavior depth indicators corresponding to each target user, and the occurrence of each target user's IP address in the same store, the possible indicators of order fraud for each target user are determined.

[0011] Based on the potential fraudulent activity indicators for all target users, real fraudulent users are selected from all target users.

[0012] Data cleaning is achieved by removing all real users who engage in fraudulent activities from the pre-acquired activity participation information of all target users.

[0013] In conjunction with the first aspect above, in one possible implementation, the step of determining the user platform usage metrics for each target user based on the platform usage data of each target user who placed an order during the official period of the monitored activity and during the pre-monitoring period of the monitored activity includes:

[0014] Get the number of times each target user logs into the target e-commerce platform, the total duration of login to the target e-commerce platform, and the total number of products viewed for each target user during each preset time period in the pre-heating period of the activity to be monitored.

[0015] Based on the number of times each target user logs into the target e-commerce platform, the total duration of logins, and the total number of products viewed within each preset time period, determine the platform local usage factor for each target user in each preset time period.

[0016] The average of the platform local usage factors of each target user during all preset time periods in the preheating period of the activity to be monitored is determined as the user platform usage index for each target user.

[0017] In conjunction with the first aspect above, in one possible implementation, determining the platform local usage factor for each target user in each preset time period based on the number of times each target user logs into the target e-commerce platform, the total duration of logins, and the total number of products viewed within each preset time period includes:

[0018] Any target user is identified as a marked user, and any preset time period during the preheating period of the activity to be monitored is identified as a marked time period;

[0019] If the marked user does not log in to the target e-commerce platform during the marked time period, the platform local usage factor of the marked user during the marked time period is set to a constant of 0.

[0020] If the marked user has logged into the target e-commerce platform during the marked time period, the ratio between the total time the marked user has logged into the target e-commerce platform during the marked time period and the total number of products viewed is determined as the marked user's product understanding index during the marked time period. The product understanding index is then multiplied by the number of times the marked user has logged into the target e-commerce platform during the marked time period, which is determined as the marked user's platform local usage factor during the marked time period.

[0021] In conjunction with the first aspect above, in one possible implementation, determining the user platform activity metrics for each target user based on the user platform usage metrics corresponding to each target user and the pre-acquired browsing data of the same category of products by each target user during the pre-heating period of the activity to be monitored includes:

[0022] Cluster all the products viewed by each target user during the preheating period of the activity to be monitored to obtain each target user's target cluster. Products within the same target cluster belong to the same product category.

[0023] The number of goods in each target cluster is determined as the number of goods represented by each target cluster;

[0024] Based on the average number of product representatives corresponding to all target clusters for each target user, and the user platform usage metrics for each target user, determine the user platform activity metrics for each target user.

[0025] In conjunction with the first aspect above, in one possible implementation, the step of determining the user behavior depth indicators for each target user based on the product inquiry and collection data of each target user during the pre-heating period of the activity to be monitored, and the user platform activity indicators corresponding to each target user, includes:

[0026] Obtain the total number of products for which each target user inquired about information, the total number of products added to their shopping cart, and the total number of products saved to their favorites during the pre-heating period of the activity to be monitored;

[0027] Based on the total number of products for which each target user consulted information, the total number of products added to their shopping cart, and the total number of products favorited during the pre-event period of the monitored activity, as well as the user platform activity metrics for each target user, we determine the in-depth user behavior metrics for each target user.

[0028] In conjunction with the first aspect above, in one possible implementation, the determination of user behavior depth indicators for each target user based on the total number of products for which each target user consulted information, the total number of products added to their shopping cart, and the total number of products favorited during the pre-heating period of the activity to be monitored, as well as the user platform activity indicators corresponding to each target user, includes:

[0029] Based on the cumulative value of the total number of products for which each target user consulted, the total number of products added to their shopping cart, and the total number of products favorited during the pre-heating period of the activity to be monitored, the product depth understanding factor corresponding to each target user is determined.

[0030] The product of the product depth understanding factor and the user platform activity index for each target user is used to determine the user behavior depth index for each target user.

[0031] In conjunction with the first aspect above, in one possible implementation, determining the potential indicators for order manipulation for each target user based on the order distribution of each target user during the official period of the monitored activity, the user behavior depth indicators corresponding to each target user, and the occurrence of IP addresses used by each target user in the same store, includes:

[0032] Based on the order distribution of each target user during the official period of the monitored activity, determine the possible factors of abnormal order generation for each target user;

[0033] Based on the number of orders placed by each target user's IP address at the same store during the official period of the monitored activity, an abnormal order quantity factor is determined for each target user.

[0034] Based on the possible factors of abnormal order generation, abnormal order quantity, and user behavior depth indicators for each target user, determine the possible indicators of order fraud for each target user.

[0035] In conjunction with the first aspect above, in one possible implementation, determining the possible factors for order generation anomalies for each target user based on the order distribution of each target user during the actual period of the monitored activity includes:

[0036] If the total number of orders placed by the target user during the official period of the monitored activity is less than or equal to the preset order quantity threshold, the abnormal order generation factor for the target user will be set to a constant of 1.

[0037] If the total number of orders placed by a target user during the official period of the monitored activity exceeds a preset order quantity threshold, the time interval between each adjacent order placed by the target user during the official period of the monitored activity is determined as the order interval duration, thus obtaining the order interval duration sequence corresponding to the target user. From the order interval duration sequence corresponding to each target user, a preset number of the smallest order interval durations are selected to form an interval duration reference group. Based on the mean and variance of the interval duration reference group corresponding to each target user, the possible factors of order generation anomalies corresponding to each target user are determined.

[0038] In conjunction with the first aspect above, in one possible implementation, determining the order quantity anomaly factor for each target user based on the number of orders placed by each target user's IP address at the same store during the official period of the monitored activity includes:

[0039] Any target user is identified as a marked user, and the IP address used by the marked user is identified as the marked IP address;

[0040] The number of orders placed by the marked IP address in the same store during the official period of the activity to be monitored is determined as the same store order quantity, and the same store order quantity set corresponding to the marked user is obtained.

[0041] The maximum value in the set of same-store order quantities corresponding to the marked user is normalized to obtain the order quantity anomaly factor corresponding to the marked user.

[0042] Secondly, the present invention provides a data cleaning system based on big data, the system comprising:

[0043] The User Platform Usage Metrics Determination Module is used to determine the user platform usage metrics for each target user based on the platform usage of each target user who places an order during the official period of the monitored activity and during the pre-heating period of the monitored activity.

[0044] The User Platform Activity Metric Determination Module is used to determine the user platform activity metric for each target user based on the user platform usage metrics for each target user and the browsing data of the same category of products for each target user during the preheating period of the activity to be monitored.

[0045] The User Behavior Depth Metrics Determination Module is used to determine the user behavior depth metrics for each target user based on the product inquiry and collection data of each target user during the pre-heating period of the activity to be monitored, as well as the user platform activity metrics corresponding to each target user.

[0046] The module for determining potential fraudulent order indicators is used to determine the potential fraudulent order indicators for each target user based on the order distribution of each target user during the official period of the monitored activity, the user behavior depth indicators corresponding to each target user, and the occurrence of each target user's IP address in the same store.

[0047] The order-brushing user filtering module is used to filter out real order-brushing users from all target users based on the possible order-brushing indicators corresponding to all target users;

[0048] The data cleaning module is used to remove all real fraudulent activity participation information from the pre-acquired activity participation information of all target users, thereby achieving data cleaning.

[0049] Thirdly, a server is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the device to perform the methods of the first aspect or any possible implementation thereof.

[0050] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0051] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0052] The present invention has the following beneficial effects:

[0053] This invention discloses a data cleaning method based on big data. By analyzing platform usage, product browsing, product inquiry and collection, and order distribution during an activity period, it quantifies multiple indicators related to the behavioral characteristics of users who engage in fraudulent transactions. These indicators include user platform usage, user platform activity, and user behavior depth. This quantifies the potential indicators of fraudulent transactions, thereby enabling relatively accurate identification of users who engage in fraudulent transactions and removing data related to genuine users, thus achieving data cleaning and improving the accuracy of data cleaning. Attached Figure Description

[0054] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of a data cleaning method based on big data according to the present invention;

[0056] Figure 2 This is a schematic diagram of the composition structure of a big data-based data cleaning system according to the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of a computer device according to the present invention. Detailed Implementation

[0058] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the specific implementation methods, structures, features, and effects of the technical solution proposed according to the present invention are described in detail below with reference to the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0060] refer to Figure 1 The diagram illustrates the flow of some embodiments of a big data-based data cleaning method according to the present invention. This big data-based data cleaning method includes the following steps:

[0061] Step S1: Based on the platform usage data of each target user who placed an order during the official period of the monitoring activity and during the pre-heating period of the monitoring activity, determine the user platform usage index corresponding to each target user.

[0062] The activity to be monitored can be a large-scale e-commerce event. The official period of the activity to be monitored can be the time period from its official start. The pre-event period can be the time period during which the activity is promoted. For example, the activity to be monitored could be a specific event, where the official period could be November 11th, and the pre-event period could be from November 1st to November 10th. Target users are those who place orders through the target e-commerce platform during the official period of the activity to be monitored. In this embodiment, users can be represented by accounts. The target e-commerce platform can be the e-commerce platform to be targeted for fraudulent order processing.

[0063] It should be noted that e-commerce platforms are online shopping platforms where users can browse products, make purchases, and pay for goods. These actions are often recorded in user logs to generate browsing and purchase records, which facilitates subsequent analysis of user platform activities and helps detect users who engage in fraudulent activities.

[0064] As an example, this step may include the following steps:

[0065] The first step is to obtain the number of times each target user logs into the target e-commerce platform, the total duration of logins, and the total number of products viewed within each preset time period during the pre-heating period of the activity to be monitored.

[0066] The preset time period can be equal to the duration of the actual activity being monitored, and can be one day. For example, if the pre-heating period for the activity being monitored is from November 1st to November 10th, and the preset time period is one day, then there can be 10 preset time periods for the pre-heating period. "Login" can represent logging in.

[0067] For example, any target user can be identified as a marked user, and any preset time period during the pre-heating period of the activity to be monitored can be identified as the marked time period. The number of times the marked user logs into the target e-commerce platform during the marked time period can be recorded as the number of times the marked user logs into the target e-commerce platform during the marked time period. The total time the marked user uses the target e-commerce platform during the marked time period can be recorded as the total time the marked user logs into the target e-commerce platform during the marked time period. The total number of products viewed by the marked user through the target e-commerce platform during the marked time period can be recorded as the total number of products viewed by the marked user during the marked time period.

[0068] The second step, based on the number of times each target user logs into the target e-commerce platform within each preset time period, the total duration of logins, and the total number of products viewed, determines the platform-specific usage factor for each target user within each preset time period. This may include the following sub-steps:

[0069] The first sub-step involves identifying any target user as a marked user and determining any preset time period within the preheating period of the aforementioned activity to be monitored as the marked time period.

[0070] The second sub-step is to set the platform local usage factor of the marked user during the marked time period to a constant 0 if the marked user does not log in to the target e-commerce platform during the marked time period.

[0071] The third sub-step involves determining the ratio between the total duration of the marked user's login to the target e-commerce platform during the marked time period and the total number of products viewed, as the product understanding index of the marked user during the marked time period. The product understanding index is then multiplied by the number of times the marked user logged into the target e-commerce platform during the marked time period, which is determined as the platform local usage factor of the marked user during the marked time period.

[0072] For example, the formula for determining the platform local usage factor of a tagged user during the tagging time period can be:

[0073] ;

[0074] Where S is the platform local usage factor of the tagged user during the tagged time period. T is the total duration of the tagged user's login to the target e-commerce platform during the tagged time period. n is the total number of products viewed by the tagged user during the tagged time period. N is the number of times the tagged user logged into the target e-commerce platform during the tagged time period. It is a metric that indicates a user's understanding of a product within a specified time period.

[0075] It should be noted that, in reality, regular users of e-commerce platforms or those who follow products closely often browse products and compare prices before and after the promotional period, hoping to purchase their desired items at lower prices. Therefore, normal users typically have a significant amount of platform usage history before the official start of the promotion. However, abnormal users engaging in fraudulent activities (such as order boosting) usually only make purchases during the promotional period itself, with very little, if any, browsing or searching history beforehand. A larger N generally indicates that the tagged user spends a relatively longer time viewing a single product within the tagged time period. This suggests that the tagged user is more likely to spend a significant amount of time viewing a single product during the pre-sale period, and more likely to have used the target e-commerce platform during this phase. In reality, users' time spent using e-commerce platforms is typically fragmented, with multiple logins per day. Therefore, a larger N generally indicates that the tagged user uses the target e-commerce platform more frequently within the tagged time period, and is more likely to be a normal user. Consequently, a larger S generally indicates that the tagged user uses the target e-commerce platform more frequently and for a relatively longer period within the tagged time period, and is more likely to be a normal user.

[0076] The third step is to determine the average of the platform local usage factors of each target user during all preset time periods in the preheating period of the activity to be monitored, and then use this average to determine the user platform usage index for each target user.

[0077] It should be noted that when the user platform usage metrics corresponding to the target user are higher, it often indicates that the target user uses the target e-commerce platform more frequently during the pre-event phase and spends more time browsing, which often suggests that the tagged user is more likely to be a normal user.

[0078] Step S2: Based on the user platform usage metrics corresponding to each target user and the browsing data of the same category of products of each target user during the preheating period of the activity to be monitored, determine the user platform activity metrics corresponding to each target user.

[0079] As an example, this step may include the following steps:

[0080] The first step is to cluster all the products that each target user browsed during the pre-heating period of the aforementioned monitoring activity to obtain each target user's corresponding target cluster.

[0081] Products within the same target cluster can belong to the same product category. Product categories can be pre-defined categories used for product classification, and may include, but are not limited to: watches, mobile phones, hair accessories, stationery, and clothing.

[0082] The second step is to determine the number of goods in each target cluster as the number of goods that represent each target cluster.

[0083] The third step is to determine the user platform activity metrics for each target user based on the average number of product representatives for all target clusters corresponding to each target user, and the user platform usage metrics for each target user.

[0084] For example, the formula for determining the user platform activity metrics corresponding to the target user can be:

[0085] ;

[0086] in, This is the user platform activity metric corresponding to the i-th target user. i is the index of the target user. It is the user platform usage metric corresponding to the i-th target user. It is the average number of product representatives corresponding to all target clusters for the i-th target user.

[0087] It should be noted that when A larger value generally indicates that the i-th target user uses the e-commerce platform more frequently and spends more time browsing during the pre-event phase, suggesting that the i-th user is more likely to be a normal user. In reality, normal users often compare the differences between similar products to improve the cost-effectiveness of their purchases—the common saying "compare three shops to avoid being ripped off." A larger value generally indicates that the i-th target user was more likely to have compared similar products during the pre-event period, and is more likely to be a normal user. Therefore, when A larger value often indicates that the i-th target user is more active during the pre-event period, and that the i-th target user is more likely to be a normal user.

[0088] In practice, if the target user does not browse products through the target e-commerce platform during the pre-heating period of the activity to be monitored, the user platform activity index for the target user can be set to 0.

[0089] Step S3: Based on the product inquiry and collection information of each target user during the pre-heating period of the activity to be monitored, and the user platform activity index corresponding to each target user, determine the user behavior depth index corresponding to each target user.

[0090] As an example, this step may include the following steps:

[0091] The first step is to obtain the total number of products that each target user inquired about, the total number of products added to their shopping cart, and the total number of products they favorited during the pre-heating period of the activity to be monitored.

[0092] For example, any target user can be identified as a marked user. The total number of products inquired about by the marked user through the target e-commerce platform during the pre-heating period of the activity to be monitored can be recorded as the total number of products inquired about by the marked user during the pre-heating period of the activity to be monitored; the total number of products added to the shopping cart by the marked user on the target e-commerce platform during the pre-heating period of the activity to be monitored can be recorded as the total number of products added to the shopping cart by the marked user during the pre-heating period of the activity to be monitored; and the total number of products collected by the marked user on the target e-commerce platform during the pre-heating period of the activity to be monitored can be recorded as the total number of products collected by the marked user during the pre-heating period of the activity to be monitored.

[0093] The second step, based on the total number of products for which each target user inquired about information, the total number of products added to their shopping cart, and the total number of products favorited during the pre-event period of the activity to be monitored, as well as the user platform activity metrics for each target user, determines the in-depth user behavior metrics for each target user. This may include the following sub-steps:

[0094] The first sub-step involves determining the product depth understanding factor for each target user based on the cumulative value of the total number of products for which they consulted, added to their shopping cart, and favorited during the pre-heating period of the activity to be monitored.

[0095] For example, the formula for determining the product understanding factor for a target user can be:

[0096] ;

[0097] in, It is the product depth understanding factor corresponding to the i-th target user. i is the index of the target user. It represents the total number of products viewed by the i-th target user during the pre-launch period of the activity to be monitored. It is the total number of products for which the i-th target user made information inquiries during the pre-heating period of the activity to be monitored. It is the total number of items added to the shopping cart by the i-th target user during the pre-heating period of the activity to be monitored. It represents the total number of items collected by the i-th target user during the pre-heating period of the activity to be monitored.

[0098] It should be noted that, in reality, compared to users who engage in fraudulent order practices, normal users often take more in-depth actions when browsing products they are interested in. For example, they might inquire about the product to gain a better understanding, add it to their shopping cart, or save it for later review or purchase. A larger value often indicates that the i-th target user is more likely to have a deeper understanding of the product during the pre-event period, and is more likely to be a normal user.

[0099] The second sub-step involves multiplying the product depth understanding factor for each target user by the user platform activity metric to determine the user behavior depth metric for each target user.

[0100] It should be noted that the higher the user behavior depth index corresponding to the target user, the more active the target user is during the pre-event period, the more likely the target user has gained a deeper understanding of the product during the pre-event period, and the more likely the target user is to be a normal user.

[0101] Step S4: Based on the order distribution of each target user during the official period of the monitoring activity, the user behavior depth indicators corresponding to each target user, and the occurrence of each target user's IP address in the same store, determine the possible indicators of order fraud for each target user.

[0102] In this invention, an IP (Internet Protocol) address identifies a specific network interface. The store in this embodiment can be a store on a target e-commerce platform.

[0103] As an example, this step may include the following steps:

[0104] The first step, based on the order distribution of each target user during the actual period of the monitored activity, is to determine the possible factors causing order generation anomalies for each target user. This may include the following sub-steps:

[0105] The first sub-step is to set the abnormal order generation factor for the target user to a constant 1 if the total number of orders placed by the target user during the official period of the monitored activity is less than or equal to the preset order quantity threshold.

[0106] The preset order quantity threshold can be a pre-set threshold, which can be 2.

[0107] In reality, there are often many fake order tasks on the day of a major e-commerce event. Therefore, a single fake order user often places a lot of orders on the day of a major e-commerce event. So, if the total number of orders placed by a target user during the official period of the event being monitored is greater than the preset order number threshold, it often indicates that the target user may be a fake order user.

[0108] The second sub-step, if the total number of orders placed by the target user during the actual period of the monitored activity exceeds the preset order quantity threshold, then determining the possible factors for abnormal order generation for each target user may include the following steps:

[0109] First, the time interval between each adjacent order placed by the target user during the actual period of the activity to be monitored is determined as the order interval duration, thus obtaining the order interval duration sequence corresponding to the target user.

[0110] Among them, the order interval duration series is a time series.

[0111] Next, a preset number of the smallest order intervals are selected from the order interval duration sequence corresponding to each target user to form an interval duration reference group.

[0112] The preset quantity can be a pre-set quantity, which can be 5. If the number of order intervals in the order interval duration sequence is less than 5, the order interval duration sequence can be directly recorded as the interval duration reference group.

[0113] Then, based on the mean and variance of the reference group for the interval duration corresponding to each target user, the possible factors for abnormal order generation for each target user are determined.

[0114] For example, if the total number of orders placed by the target user during the actual period of the monitored activity exceeds a preset order quantity threshold, the formula for determining the possible factors of abnormal order generation for the target user can be:

[0115] ;

[0116] in, It represents the possible factors for order generation anomalies corresponding to the i-th target user. i is the index of the target user. It is an exponential function with the natural constant as its base. It is the variance of the interval duration of all orders in the interval duration reference group corresponding to the i-th target user. It is the average of the interval durations of all orders in the reference group corresponding to the i-th target user.

[0117] It should be noted that in reality, users who engage in fraudulent order placement often do not need to understand the product details. The operational procedures for different products are often the same, therefore, the time intervals between adjacent orders are often similar. Secondly, compared to normal users, users who engage in fraudulent order placement are often more proficient in operating the platform's products. Furthermore, after completing one order, they often do not need to understand the product details for the next order and often directly place the next purchase. Therefore, the time intervals between adjacent orders for users who engage in fraudulent order placement are often shorter. Therefore, when The larger the value, the shorter the time interval between adjacent orders of the i-th target user, and the less similar the time intervals between adjacent orders, which often indicates that the i-th target user is more likely to be a fraudulent order user.

[0118] The second step, based on the number of orders placed by each target user's IP address at the same store during the official period of the monitored activity, determines the abnormal order quantity factors for each target user, which may include the following sub-steps:

[0119] The first sub-step involves identifying any target user as a marked user and determining the IP address used by the marked user as the marked IP address.

[0120] The second sub-step involves determining the number of orders placed by the aforementioned marked IP addresses in the same store during the actual period of the monitored activity as the same store order quantity, thus obtaining the same store order quantity set corresponding to the aforementioned marked users.

[0121] The third sub-step is to normalize the maximum value in the set of same-store order quantities corresponding to the above-mentioned marked users to obtain the order quantity anomaly factor corresponding to the above-mentioned marked users.

[0122] In reality, when users place orders for selected products on a platform, the platform often records the user's IP address. During promotional periods, a large number of users participate in online shopping. Under normal circumstances, online shoppers from a single store tend to be dispersed and random. However, in large-scale order-brushing operations, the operations are often carried out by the same order-brushing studio. These studios use a limited number of network exits (IP addresses) to operate a large number of order-brushing accounts, resulting in multiple different buyer users coming from the same or a few IP addresses. Therefore, the larger the anomaly factor of the number of orders corresponding to a marked user, the more likely the marked user's IP address is to belong to an order-brushing studio, and the more likely the marked user is to be a member of an order-brushing studio.

[0123] The third step is to determine the possible indicators for order fraud for each target user based on the abnormality factors, abnormal order quantity factors, and user behavior depth indicators generated for each target user.

[0124] For example, the formula for determining the potential metrics for fraudulent orders corresponding to a target user can be:

[0125] ;

[0126] in, This represents the potential metrics for fraudulent transactions corresponding to the i-th target user. i is the index of the target user. It is a normalization function. It is the possible factor for abnormal order generation corresponding to the i-th target user. It is the abnormal factor of the number of orders corresponding to the i-th target user. It is an exponential function with the natural constant as its base. It is the user behavior depth metric corresponding to the i-th target user.

[0127] It should be noted that when A larger value generally indicates that the time interval between adjacent orders of the i-th target user is relatively shorter, and the time intervals between adjacent orders are relatively less similar, which often suggests that the i-th target user is more likely to be a fraudulent order user. A larger value generally indicates that the IP address of the i-th target user is more likely to belong to a fraudulent order-boosting studio, and vice versa. A larger value generally indicates that the i-th target user was more active during the pre-event period, more likely to have a deeper understanding of the product, and more likely to be a typical user. Therefore, when... The larger the value, the more likely the i-th target user is to be a fraudulent user.

[0128] Step S5: Based on the potential fraudulent activity indicators for all target users, filter out the real fraudulent users from all target users.

[0129] As an example, if the potential fraudulent order metrics for a target user exceed a preset fraudulent order threshold, the target user is determined to be a genuine fraudulent order user. The preset fraudulent order threshold can be a pre-set threshold, such as 0.6.

[0130] Step S6: Remove all real fraudulent activity participation information from the pre-acquired activity participation information of all target users to be monitored, thereby achieving data cleaning.

[0131] Among them, the participation information of the target user in the monitored activity can represent the data recorded by the target user through the target e-commerce platform in the monitored activity.

[0132] refer to Figure 2 Based on the same inventive concept as the above-described method embodiments, this invention provides a data cleaning system based on big data. The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the computer program implements the steps of a data cleaning method based on big data, specifically including:

[0133] The User Platform Usage Metrics Determination Module 201 is used to determine the user platform usage metrics for each target user based on the platform usage of each target user who places an order during the official period of the monitored activity and during the preheating period of the monitored activity.

[0134] The user platform activity indicator determination module 202 is used to determine the user platform activity indicator for each target user based on the user platform usage indicator for each target user and the browsing data of the same category of products of each target user during the preheating period of the activity to be monitored.

[0135] The user behavior depth indicator determination module 203 is used to determine the user behavior depth indicator for each target user based on the product inquiry and collection information of each target user during the preheating period of the activity to be monitored, as well as the user platform activity indicator corresponding to each target user.

[0136] The module 204 for determining potential fraudulent order indicators is used to determine the potential fraudulent order indicators for each target user based on the order distribution of each target user during the official period of the monitored activity, the user behavior depth indicators corresponding to each target user, and the occurrence of the IP address used by each target user in the same store.

[0137] The order-brushing user screening module 205 is used to filter out real order-brushing users from all target users based on the possible order-brushing indicators corresponding to all target users.

[0138] The data cleaning module 206 is used to remove all real fraudulent activity participation information from the pre-acquired activity participation information of all target users, thereby achieving data cleaning.

[0139] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. For example, as shown... Figure 3 As shown, the computer device 300 includes: a memory 301, a processor 302, and a computer program 303 stored in the memory 301 and running on the processor 302, wherein when the processor 302 executes the computer program 303, the computer device can execute any of the data cleaning methods based on big data described above.

[0140] Based on the same inventive concept as the above-described method embodiments, the present invention provides a server, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, enabling the device to execute any of the above-described data cleaning methods based on big data.

[0141] Based on the same inventive concept as the above-described method embodiments, the present invention provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to execute any of the above-described big data-based data cleaning methods.

[0142] Based on the same inventive concept as the above-described method embodiments, the present invention provides a computer-readable storage medium storing computer program code, which, when run on a computer, causes the computer to execute any of the above-described data cleaning methods based on big data.

[0143] In summary, this invention quantifies multiple indicators related to the behavioral characteristics of users engaging in fraudulent activities by analyzing platform usage, product browsing, product inquiries and collections, and order distribution during the activity period. These indicators include user platform usage indicators, user platform activity indicators, and user behavior depth indicators. This quantifies the potential fraudulent activity indicators that characterize the overall likelihood of users engaging in fraudulent activities, thereby achieving relatively accurate identification of fraudulent users and removing data related to genuine fraudulent users, thus improving the accuracy of data cleaning.

[0144] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A big data based data cleaning method, characterized in that, Includes the following steps: The monitored activities are divided into two specific phases: a warm-up period and a formal period. The warm-up period is a specified time before the official start of the activity, and the formal period is the core trading period after the official start of the activity. Based on the platform usage data of each target user who placed an order during the official period of the monitored activity, obtained in advance, the platform usage metrics for each target user are determined. The platform usage data includes the number of times each target user logs into the target e-commerce platform, the total duration of login to the target e-commerce platform, and the total number of products viewed during each preset time period in the pre-monitoring period of the monitored activity. Based on the user platform usage metrics corresponding to each target user and the browsing data of the same category of products of each target user during the pre-heating period of the activity to be monitored, the user platform activity metrics corresponding to each target user are determined, including: clustering all products browsed by each target user during the pre-heating period of the activity to be monitored to obtain each target user's target cluster, wherein products in the same target cluster belong to the same product category; The number of goods in each target cluster is determined as the number of goods represented by each target cluster; Based on the average number of product representatives corresponding to all target clusters for each target user, and the user platform usage metrics for each target user, determine the user platform activity metrics for each target user. Based on the product inquiries and collections of each target user during the pre-heating period of the activity to be monitored, and the user platform activity indicators corresponding to each target user, the user behavior depth indicators corresponding to each target user are determined. Based on the order distribution of each target user during the official period of the monitored activity, the user behavior depth indicators corresponding to each target user, and the occurrence of each target user's IP address in the same store, the possible indicators of order fraud for each target user are determined. Based on the potential fraudulent activity indicators for all target users, real fraudulent users are selected from all target users. Data cleaning is achieved by removing all real users who engage in fraudulent activities from the pre-acquired activity participation information of all target users.

2. The big data based data cleaning method of claim 1, wherein, The method involves determining the platform usage metrics for each target user based on the platform usage data of each target user who placed an order during the pre-monitoring period of the activity, obtained in advance. This includes: Based on the number of times each target user logs into the target e-commerce platform, the total duration of logins, and the total number of products viewed within each preset time period, determine the platform local usage factor for each target user in each preset time period. The average of the platform local usage factors of each target user during all preset time periods in the preheating period of the activity to be monitored is determined as the user platform usage index for each target user.

3. The big data based data cleaning method of claim 2, wherein, The method of determining the platform local usage factor for each target user in each preset time period based on the number of times each target user logs into the target e-commerce platform, the total duration of login to the target e-commerce platform, and the total number of products viewed within each preset time period includes: Any target user is identified as a marked user, and any preset time period during the preheating period of the activity to be monitored is identified as a marked time period; If the marked user does not log in to the target e-commerce platform during the marked time period, the platform local usage factor of the marked user during the marked time period is set to a constant of 0. If the marked user has logged into the target e-commerce platform during the marked time period, the ratio between the total time the marked user has logged into the target e-commerce platform during the marked time period and the total number of products viewed is determined as the marked user's product understanding index during the marked time period. The product understanding index is then multiplied by the number of times the marked user has logged into the target e-commerce platform during the marked time period, which is determined as the marked user's platform local usage factor during the marked time period.

4. The big data based data cleaning method of claim 1, wherein, The method involves determining in-depth user behavior metrics for each target user based on pre-acquired product inquiries and saves during the pre-heating period of the activity to be monitored, and user platform activity metrics for each target user. These metrics include: Obtain the total number of products for which each target user inquired about information, the total number of products added to their shopping cart, and the total number of products saved to their favorites during the pre-heating period of the activity to be monitored; Based on the total number of products for which each target user consulted information, the total number of products added to their shopping cart, and the total number of products favorited during the pre-event period of the monitored activity, as well as the user platform activity metrics for each target user, we determine the in-depth user behavior metrics for each target user.

5. The big data based data cleaning method of claim 4, wherein, The method determines the user behavior depth indicators for each target user based on the total number of products for which each target user inquired about information, the total number of products added to their shopping cart, and the total number of products favorited during the pre-heating period of the activity to be monitored, as well as the user platform activity indicators for each target user. These indicators include: Based on the cumulative value of the total number of products for which each target user consulted, the total number of products added to their shopping cart, and the total number of products favorited during the pre-heating period of the activity to be monitored, the product depth understanding factor corresponding to each target user is determined. The product of the product depth understanding factor and the user platform activity index for each target user is used to determine the user behavior depth index for each target user.

6. The big data based data cleaning method of claim 1, wherein, The method determines potential fraudulent order indicators for each target user based on the order distribution of each target user during the official period of the monitored activity, the user behavior depth indicators corresponding to each target user, and the occurrence of each target user's IP address in the same store. These indicators include: Based on the order distribution of each target user during the official period of the monitored activity, determine the possible factors of abnormal order generation for each target user; Based on the number of orders placed by each target user's IP address at the same store during the official period of the monitored activity, an abnormal order quantity factor is determined for each target user. Based on the possible factors of abnormal order generation, abnormal order quantity, and user behavior depth indicators for each target user, determine the possible indicators of order fraud for each target user.

7. The big data based data cleaning method of claim 6, wherein, The step of determining possible factors for order generation anomalies for each target user based on the order distribution of each target user during the official period of the monitored activity includes: If the total number of orders placed by the target user during the official period of the monitored activity is less than or equal to the preset order quantity threshold, the abnormal order generation factor for the target user will be set to a constant of 1. If the total number of orders placed by a target user during the official period of the monitored activity exceeds a preset order quantity threshold, the time interval between each adjacent order placed by the target user during the official period of the monitored activity is determined as the order interval duration, thus obtaining the order interval duration sequence corresponding to the target user. From the order interval duration sequence corresponding to each target user, a preset number of the smallest order interval durations are selected to form an interval duration reference group. Based on the mean and variance of the interval duration reference group corresponding to each target user, the possible factors of order generation anomalies corresponding to each target user are determined.

8. The big data based data cleaning method of claim 6, wherein, The step of determining the order quantity anomaly factor for each target user based on the number of orders placed by each target user's IP address at the same store during the official period of the monitored activity includes: Any target user is identified as a marked user, and the IP address used by the marked user is identified as the marked IP address; The number of orders placed by the marked IP address in the same store during the official period of the activity to be monitored is determined as the same store order quantity, and the same store order quantity set corresponding to the marked user is obtained. The maximum value in the set of same-store order quantities corresponding to the marked user is normalized to obtain the order quantity anomaly factor corresponding to the marked user. 9.A big data based data cleaning system, characterized in that, It includes a processor and a memory, the processor being used to process instructions stored in the memory to implement a data cleaning method based on big data as described in any one of claims 1-8.

Images