E-commerce data automatic acquisition method and system

Abstract

The invention belongs to the technical field of data processing, and provides an e-commerce data automatic acquisition method and system, and the method specifically comprises the steps: firstly, building an e-commerce credit extension feature database with a natural day as an acquisition granularity, constructing a stage coverage rate vector for each credit extension feature, then constructing an access condition, screening and comparing merchants from a stock library, and carrying out the collection of a plurality of credit extension features; and finally, constructing a cold start offset identification model according to the stage coverage rate vectors of the comparison merchant and the to-be-evaluated merchant, obtaining an offset mark, and shielding the e-commerce data collected in the time period of triggering the offset mark. According to the method, the time point of the abnormal comparison ability of the constructed portrait in the new business opportunity operation process is depicted through portrait cold start offset, so that the acquisition quality of portrait construction data is improved on the premise of not significantly prolonging the evaluation period, the accuracy and risk discrimination of platform merchant loan decision are further improved, and the business opportunity evaluation efficiency is improved. And the credit granting risk caused by mismatching of portraits in the face of novice merchants is reduced.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data processing, and particularly relates to an e-commerce data automatic collection method and system. BACKGROUND

[0002] An e-commerce platform with a merchant loan function needs to improve the credit efficiency under the premise of ensuring the safety of funds, so the platform usually needs to construct a merchant portrait based on the operating data of the merchant, and then quantitatively evaluate the operating ability of the merchant. The existing method often relies on long-period transaction flow, order volume, return rate and other time series data to accurately depict the real operating ability and repayment ability of the merchant. However, the life cycle of e-commerce merchants is highly uneven, and some merchants have a short duration, while potential merchants have a strong demand for funds at the initial stage of entry. If the evaluation period is lengthened, potential new merchant customers may be lost due to insufficient response speed of credit decision-making.

[0003] Therefore, the existing technology has realized the rapid construction of a short-term portrait of a merchant to be evaluated based on short observation windows of operating data, and comparison with the historical portrait of the inventory merchant to speed up the credit decision-making. However, the merchant portrait constructed by short-period data generally has a portrait cold start bias problem in the novice period, because the core indicators of the merchant are often simultaneously affected by shop building, product structure adjustment, pricing exploration, activity concentrated investment and supply chain adaptation in the initial settlement period, showing significant fluctuations and structural bias. The statistical distribution is inevitably systematically misplaced compared with the long-term distribution of the merchant after entering the stable operating stage.

[0004] The portrait cold start bias problem is essentially caused by the mismatch of the sample time window, because the data segment used to depict the short-term portrait of the merchant to be evaluated is mainly in the operating exploration stage, while the inventory merchant portrait used for reference is mostly aggregated from the data of the stable operating stage. The two are not equivalent in terms of operating state, traffic structure and activity intensity, so when the platform directly compares the similarity between the short-period portrait containing the cold start drift component and the portrait of the inventory merchant without distinction, it will significantly amplify the influence of the stage mismatch. The most significant two polar reactions include, one is a high-potential new merchant in the normal trial period but with large fluctuations in indicators, which is easily mis-matched to a high-risk merchant group and over-conservatively granted credit; the other is a merchant relying on short-term subsidy impulse or abnormal diversion, which may be mis-matched to a high-quality merchant group, and its real and sustainable operating ability is hidden, resulting in systematic bias of the credit model. Therefore, there is an urgent need for an e-commerce data classification method from the perspective of the platform. SUMMARY

[0005] The purpose of this invention is to propose an automated e-commerce data collection method and system to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0006] To achieve the above objectives, according to one aspect of the present invention, an automated e-commerce data collection method is provided, the method comprising the following steps: S100 establishes an e-commerce credit feature database with natural days as the collection granularity, which includes several credit features; S200, constructs a phase coverage vector for each credit feature; S300 establishes admission criteria to screen and compare merchants from the existing database; S400: Construct a cold start offset recognition model based on the phase coverage vectors of the compared merchants and the merchants to be evaluated, and obtain offset markers; S500 performs masking on e-commerce data collected during the period of trigger offset marking.

[0007] Furthermore, in step S100, an e-commerce credit feature database is established with natural days as the collection granularity. The method for including several credit features is as follows: the e-commerce credit feature database is used to store the corresponding values ​​of each credit feature obtained by each merchant. A set of credit features is preset in the database. The set of credit features includes one or more of the following: transaction scale features, transaction quality features, operating efficiency features, and capital recovery features. Each natural day is used as an observation point, and the corresponding value of the credit feature is obtained at each observation point.

[0008] Further, in step S200, the method for constructing the stage coverage vector for each credit feature is as follows: obtain the observation duration of the merchant to be evaluated, extract each credit feature within the current observation duration for each merchant, and record the sequence of values ​​of any credit feature within the current observation duration as the feature observation sequence. Normalize the feature observation sequence, and record the average of each maximum value of the feature observation sequence as the peak level. When an element in the feature observation sequence is larger than all the elements in its reverse time direction, mark the element as the coverage peak. The proportion of the coverage peak in the feature observation sequence is recorded as the coverage level. The product of the peak level and the coverage level is recorded as the coverage rate of the credit feature. The stage coverage vector for the current natural day is constructed from the coverage rates of each credit feature.

[0009] The Dur of observation for evaluating a merchant refers to the number of natural days between the time point when the merchant's credit characteristics were first recorded and the current time, with a constraint that Dur ≥ 30 natural days; the current observation duration refers to each natural day in the Dur period from the current time to its reverse time direction.

[0010] The phase coverage is to encode whether the feature distribution is stable, whether it is frequently refreshed to a new high, and whether it is in a momentum period into a low-dimensional vector, so as to facilitate the next step to clearly distinguish between non-stable momentum behavior and stable period regular fluctuation behavior, eliminate the scale difference between different merchants in the subsequent program, highlight the frequently refreshed new high feature of cold start offset behavior in the time series, and compress each natural day into a phase stability index, which can significantly enhance the recognition sensitivity of the model to whether the business state at the time point is in an abnormal rising or abnormal fluctuation period when performing time series analysis.

[0011] Further, in step S300, the method of screening and comparing the merchant from the inventory bank is: the inventory bank is a set of each merchant in the e-commerce credit feature database, and the access conditions include: at least completing a merchant loan in the platform, having at least 90 days of valid operation records, and the category range being the same as or belonging to the same category cluster as the merchant to be evaluated; from the inventory bank, the merchants meeting each access condition are taken as comparison merchants; The specific requirement of at least completing a merchant loan in the platform includes: the platform maintains a merchant loan record table for each merchant, in which, for each merchant loan, the application time, approval result, credit limit, loan amount, repayment plan and repayment status information are recorded. If the merchant has at least one record with a loan amount greater than zero and a loan state of having been loaned in the loan record table, and the state of the loan is normal repayment or has been cleared, it is determined that the merchant has at least completed a merchant loan in the platform.

[0012] The completion degree of the merchant loan limitation makes the inventory merchants entering the comparison merchant set all experience a complete loan credit, loan and repayment process on the platform, and the platform can evaluate the default risk based on the historical loan performance, so as to ensure that the comparison merchants are representative in the credit behavior dimension.

[0013] Having at least 90 days of valid operation records means that in a 180-day observation period, the merchant counts the operation of each natural day, and the number of valid operation days exceeds 90 natural days to meet the condition, wherein when the natural day meets one of the following conditions, the natural day is recorded as the valid operation day of the merchant: a) There is a paid order in the natural day, and the number of transaction orders in the natural day is greater than zero; b) At least one transaction size feature and one operating efficiency feature of the main trusted features of the natural day have legal values, wherein the transaction size feature is the total transaction amount or the number of payment orders, and the operating efficiency feature can be the number of visitors or the payment conversion rate; it is known that if there are abnormal records with missing time stamps, negative amounts or significantly exceeding the preset value range in the natural day corresponding trusted feature records, the natural day is not counted as an effective operating day.

[0014] Through the effective operating record limitation, not only is there actual transaction or effective access behavior, but also the data used for trust evaluation is guaranteed to have basic integrity by removing abnormal records.

[0015] The category range is the same as or belongs to the same category cluster as the merchant to be evaluated, which means that the e-commerce platform pre-sets the corresponding category code for each item for sale according to its own commodity classification system, and the category code is divided according to the hierarchical structure. Based on the transaction amount or the number of transaction orders of each category of each merchant within a preset observation period, the platform calculates the contribution proportion of each category, and identifies one or more categories with the highest contribution proportion as the main category of the merchant.

[0016] Among them, the platform pre-aggregates multiple sub-categories into several category clusters according to business similarity, for example, it can attribute multiple secondary categories such as men's clothing, women's clothing, and children's clothing to a primary category of clothing category cluster, attribute multiple secondary categories such as mobile phones, tablets, and computers to a digital category cluster, and attribute multiple secondary categories such as makeup and skincare to a primary category of beauty and care category cluster. The category cluster to which the main category of each merchant belongs is the category cluster label defined by the merchant.

[0017] When selecting and comparing merchants from the inventory library, the main category of the merchant to be evaluated and its corresponding category cluster label are obtained; among the candidate merchants, the merchants whose main category is the same as that of the merchant to be evaluated, or whose main category is not the same but belongs to the same category cluster label as that of the merchant to be evaluated, are selected as merchants whose category range is the same as or belongs to the same category cluster as the merchant to be evaluated. Preferably, for merchants operating multiple categories and having no obvious main category, the category cluster with the highest contribution proportion can be selected as the main category cluster of the merchant for comparison according to the contribution proportion.

[0018] Through the category range limitation, the selected inventory merchants for comparison have a basic consistent business direction in terms of product type, price band, and target user group as the merchant to be evaluated, avoiding mixing comparison of merchants with completely different business types and introducing additional interference.

[0019] Further, in step S400, a cold start offset identification model is constructed according to the comparison of the phase coverage rate vectors of the merchant and the merchant to be evaluated, and the method of obtaining the offset label is: the phase coverage rate vectors of the same natural day and different comparison merchants are taken as a column, and the phase coverage rate vectors of the same comparison merchant and different natural days are taken as a row to construct a matrix, which is recorded as a cold start offset identification model; for any natural day, the Mahalanobis distance of the phase coverage rate vectors of the merchant to be evaluated and any comparison merchant is recorded as a cold start offset index, and the median and standard deviation of all cold start offset indexes are recorded as a cold start offset median and an offset fluctuation, respectively. The quotient of the difference between the cold start offset index and the cold start offset median after the difference is calculated and the offset fluctuation is recorded as a benchmark offset index; for any comparison merchant, the ratio of the number of natural days with the benchmark offset index greater than 0 and less than 0 is recorded as a comparison coefficient, and if the comparison coefficient is greater than or equal to 0.5, the comparison merchant is recorded as a high contrast comparison merchant, otherwise it is recorded as a low contrast comparison merchant; the offset labeling in this step is based on the cold start offset identification model.

[0020] The design of the benchmark offset index is an effective fitting of the non-inertial system relative motion phenomenon in the real world. The prior art usually directly uses the absolute numerical distance to measure the difference between the merchant to be evaluated and the comparison merchant. This method ignores the systematic drift of the reference system itself, that is, it fails to eliminate the overall data offset of the comparison merchant group due to seasonal market fluctuations, industry market trends, and non-characteristic data jitter caused by daily order randomness. This method calculates the difference between the cold start offset index and the median and divides it by the offset fluctuation, which is essentially equivalent to constructing a dynamic adaptive normalization filter. The index maps the behavior data of the merchant to be evaluated to a dimensionless tensor space with the group common feature as the zero point and the group fluctuation distribution rate as the unit scale. This processing method is similar to introducing a signal-to-noise ratio weighting mechanism in data processing. By structural constraint, the model eliminates common mode interference caused by industry cycles or industry cycles, so that subsequent operations only solve the intrinsic anomalies caused by individual business strategy adjustments, effectively excluding random background noise interference that does not represent real business state changes in business logic.

[0021] For any natural day, record all the benchmark offset indexes of high contrast ratio against merchants and low contrast ratio against merchants as high contrast offset vector and low contrast offset vector respectively; perform Kmeans clustering analysis on the high contrast offset vector and the low contrast offset vector of all natural days respectively, wherein the category selection of the clustering analysis is realized through a scatter plot, and the KMeans function of the scikit-learn library in python is used to realize the Kmeans clustering analysis; if the number of samples in a cluster is less than the average number of samples in a cluster, the cluster is recorded as a collapsed cluster; wherein the number of samples in a cluster refers to the number of elements in each cluster obtained by the clustering analysis, and the average number of samples in a cluster is the average of the number of samples corresponding to each cluster; the process of Kmeans clustering analysis of the high contrast offset vector and the low contrast offset vector is independent of each other, so the high contrast offset vector and the low contrast offset vector respectively constitute a collapsed cluster.

[0022] The introduction of the collapsed cluster is a critical state definition of the topological structure of the data space. This process refers to the characteristic setting that small clusters are often regarded as calculation residues in traditional clustering analysis, and the present application defines it as a representation of failure of operating state phase transition. Normal business behavior should form a stable data aggregation area with high sample density in the feature space, that is, a statistically significant business portrait is constructed through continuous and stable daily traffic conversion and order fulfillment behavior; and the collapsed cluster accurately captures the discrete data state that cannot form a stable structure due to lack of time continuity or consistent behavior. This method imposes a minimum entropy reduction constraint at the algorithm level through the definition of the collapsed cluster, that is, it forces business data to have the ability to reduce the degree of system disorder to be considered as an effective feature, thereby separating non-steady-state behaviors such as exploratory promotion, pulse type single, or short impulse specific to the cold start period from the stable business portrait, accurately identifying transient abnormal periods with insufficient statistical confidence and unable to represent real long-term business capabilities, and thereby improving the accuracy of offset labeling.

[0023] In the collapsed cluster of the low contrast offset vector clustering result, identify whether there is a time-continuous Lth natural day or more, if there is, mark it as a pre-offset; mark the natural day with a pre-offset in the collapsed cluster of the high contrast offset vector clustering result.

[0024] Wherein Lth is a preset threshold of the number of natural days, the default value is 3, the minimum value is 3, and the upper limit is 0.25 times the time length of the matrix.

[0025] Since the above benchmark offset index is obtained by relying on the cold start offset median and the offset fluctuation, by calculating the Mahalanobis distance of the phase coverage vector of the to-be-evaluated merchant and any comparison merchant, the covariance structure of the data can be effectively utilized, the anomaly can be better measured, and at the same time, in order to make the Mahalanobis distance of different natural days comparable, the cold start offset median and the offset fluctuation are standardized, so that the benchmark offset index can be used to compare the data of the to-be-evaluated merchant and the comparison merchant on the same natural day, and the index offset caused by the cold start can be effectively identified, but since the cold start offset is detected only by relying on the benchmark offset index of each independent natural day, the autocorrelation and trend on the time series cannot be completely captured; The abnormal points may appear continuously in time, causing some normal and interval far natural days to be misjudged as abnormal, and therefore the trend fluctuation of the time series needs to be further considered as a whole, in order to improve the robustness and accuracy of the determination, the application proposes a more preferred scheme; Further, in step S400, a cold start offset identification model is constructed according to the phase coverage vectors of the comparison merchants and the to-be-evaluated merchant, and the offset marking method is: taking the phase coverage vectors of different comparison merchants on the same natural day as a row, and taking the phase coverage vectors of different natural days of the same comparison merchant as a column to construct a matrix, which is denoted as a cold start offset identification model: Based on the full time period coverage vector sequence, the dynamic time warping distance DL of each natural day to-be-evaluated merchant and all comparison merchants is calculated, wherein the dynamic time warping distance is calculated by using the distance function of the dtaidistance library in python; the comparison merchants are divided into sk behavior groups by using the Kmeans algorithm, wherein sk is the number of clusters in the clustering, and the value range of sk is sk [5, 50], and the value of sk is adjusted according to the clustering effect; the average value of the DL of each behavior group is calculated, the to-be-evaluated merchant is classified into the behavior group with the smallest average value of DL, to form a time series resonance cluster, and the set of other comparison merchants in the time series resonance cluster except the to-be-evaluated merchant is defined as a same kind reference group; The construction of the time series resonance cluster is a modeling of the non-synchronous coupling phenomenon in the e-commerce operation data. Different merchants respond to the same market signal, such as a network-wide promotion or seasonal replacement, and there is an inherent phase lag. Even the merchants of the same category will also be limited by the supply chain inventory speed and the difference in operation strategy, so that the traffic wave peak often appears out of position on the time axis, for example, merchant A bursts on Monday, and merchant B bursts on Wednesday due to inventory delay.

[0026] The prior art usually directly uses the Euclidean distance to hard cluster the merchant to be evaluated with the sample library, which microcosmically assumes that the operating rhythm of all merchants must be strictly synchronized, resulting in that those merchants with consistent operating logic but phase lag are incorrectly determined as different species. Although the method uses the KMeans algorithm based on the Euclidean distance to perform feature dimension reduction when constructing the basic behavior group, extracts several mainstream operating forms existing in the market, but in the core classification decision link, the method introduces dynamic time warping as a higher priority measurement standard, thereby establishing an elastic time sequence connection, so that even if the coverage rate vector of the merchant to be evaluated is far away from a behavior group in the Euclidean space due to time dislocation, as long as the two present the minimum path cost in the DTW measurement, the model will still bind them as a time sequence resonance cluster. This hybrid architecture actually builds a time axis adaptive calibrator inside the algorithm to ensure that the selection of the reference system is based on the isomorphism of the operating essence, rather than rigid time point alignment, which is conducive to improving the inclusiveness of the model for the same frequency different step phenomenon in the real e-commerce scene.

[0027] The time sequence resonance cluster is essentially a topological manifold that tolerates elastic deformation, which forces the model to ignore non-critical time axis stretching and only focuses on the same frequency fluctuations of the coverage rate vector in the form. This enables the subsequent steps to capture the process of the abnormal behavior of the merchant to be evaluated that truly deviates from the group commonality more accurately on a physically meaningful reference surface.

[0028] For all merchants in the time sequence resonance cluster, the modulus of the coverage rate vector change amplitude relative to the previous natural day is calculated every natural day, denoted as DF. The modulus DF of the coverage rate vector change amplitude is used to measure the overall change degree of the same merchant between adjacent natural days. For any merchant in the time sequence resonance cluster, the difference between the current natural day's stage coverage rate vector and the previous natural day's stage coverage rate vector is subtracted by component on a certain natural day, to obtain a coverage rate change vector. Then the square sum of the difference values of each component of the coverage rate change vector is summed and the square root is taken, and the result is defined as the modulus DF of the coverage rate vector change amplitude relative to the previous natural day. The average value of DF of all merchants on the same day is DF average, and the natural day whose DF average exceeds the upper quartile value is identified by using the box plot and recorded as the universal mutation point; If in the universal mutation point, if the DF of the merchant to be evaluated exceeds the DF average, it is recorded as an abnormally high frequency point, otherwise it is a potential high frequency point. If in the non-universal mutation point, if the DF of the merchant to be evaluated exceeds the DF average, it is recorded as a potential abnormal point, otherwise it is an abnormally low frequency point; The center trend vector of the same kind reference group in each natural day is calculated, and the calculation method of the center trend vector is: the arithmetic mean value of the vector of each dimension of all merchants in the same kind reference group is calculated respectively, and the arithmetic mean value is combined according to the original order (stage coverage rate vector corresponds to the order) to form a new vector, that is, the center trend vector of the natural day; the cosine similarity GcSim between the stage coverage rate vector of the merchant to be evaluated and the center trend vector is calculated, and a similarity time sequence is constructed, and any natural day and the previous Num natural days form a similarity observation window; wherein Num is a preset integer variable, and the value range of Num is Num∈[5, 10], the moving average GuEMA and the standard deviation GoErr of the similarity observation window are calculated, and if the natural day satisfies GcSim≤GuEMA-2GoErr, it is determined that it is a mode deviation anomaly; The range of the coverage rate vector is defined as the stage difference Cs, the sum of the coverage rates of the merchants in the same kind reference group is recorded as the total coverage Cv, and the coverage consistency CI is calculated according to the total coverage: CI=|Cv-Cs| / ln(1+Cs+GcSim^2); if the natural day satisfies that the CI of the merchant to be evaluated is less than half of the average CI of all merchants, it is determined that it is a dispersion anomaly; wherein ln() represents the logarithmic function with e as the base number; The setting of the coverage consistency is to solve the false alarm problem caused by the phenomenon that the larger the data volume is, the more violent the natural fluctuation is. Since the daily order quantity of the head merchant is huge in the real e-commerce scene, its normal random fluctuation may far exceed the total quantity of small and medium-sized merchants in an absolute value. If only a linear numerical threshold is used, large merchants are easily misjudged as abnormal. This method constructs a dynamic fluctuation tolerance regulator through a formula: a logarithmic compression mechanism is used to realize desensitization of large numbers, and as the business scale and fluctuation range of the merchant increase exponentially, the denominator increases, thereby compressing the absolute difference value on the numerator, and preventing the index from being too large due to the general surge in traffic during the big promotion.

[0029] The square term of the cosine similarity is taken as a trend alignment factor and coded into the damping term of the denominator, and the principle is to establish a kind of follow-up compliance exemption mechanism: when the traffic curve of the merchant to be evaluated is highly similar to the industry index, the denominator increases significantly, and the calculated CI abnormal value is strongly suppressed. Therefore, as long as the rise and fall rhythm of the merchant conforms to the trend of the industry index, for example, the violent surge during Double Eleven, the evaluation standard for the numerical violent fluctuation is relaxed. The CI index effectively establishes a dynamic balance between the numerical absolute deviation and the trend similarity, forces the model to exclude those pseudo anomalies with huge numerical fluctuations but conforming to the evolution law of the industry index, so as to ensure that the dispersion anomaly finally locked is the out-of-order behavior that deviates from the normal operation logic trend.

[0030] When the natural day is an abnormal high frequency point, the offset label is marked; when the natural day is an abnormal low frequency point, it is determined as a non-abnormal point, when the natural day is a potential high frequency point and at least one of the mode deviation anomaly or the dispersion anomaly is satisfied, the offset label is marked; when the natural day is a potential abnormal point, and at the same time, the mode deviation anomaly and the dispersion anomaly are satisfied, the offset label is marked.

[0031] Beneficial effect: Since the offset label is obtained according to the coverage vector time series analysis of the to-be-evaluated merchant and the comparison merchant, the time point at which the profile cold start offset causes the profile constructed in the new business operation process to appear abnormal comparison ability can be effectively described, thereby providing a basis for further removing the non-steady-state business data samples formed in the cold start stage of the new merchant, improving the accuracy of the credit profile evaluation result, providing a high-quality observation interval for accurately matching the comparison profile data of the to-be-evaluated merchant and the inventory merchant, and further optimizing the quality of the e-commerce business data collected for subsequent credit evaluation.

[0032] Further, in step S500, the method of shielding the e-commerce data collected in the offset label period is: obtaining the offset label set of the to-be-evaluated merchant in step, marking the e-commerce data corresponding to the natural day in the offset label set as shielding data, and ignoring the shielding data when constructing the credit profile or calculating the credit evaluation index for the to-be-evaluated merchant in the subsequent, and generating the credit evaluation result based only on the e-commerce data corresponding to the natural day not belonging to the offset label time set.

[0033] For the to-be-evaluated merchant, the offset label time set is used to record the natural day that is determined by the cold start offset recognition model as having obvious cold start offset in the observation time length. Each element in the offset label time set corresponds to a natural day or a continuous natural day interval.

[0034] The method introduces an automatic shielding mechanism for the offset label time period at the end of the e-commerce data automatic collection link, so that when receiving a credit evaluation request, the e-commerce data corresponding to the cold start offset period can be automatically excluded in the data reading stage, and only the natural day data not marked as offset is used as the sample basis for credit evaluation, thereby reducing the interference of behaviors such as novice period abnormal impulse, short-term subsidy stacking, and abnormal diversion on the credit profile, and improving the comparison ability of the credit evaluation result.

[0035] Preferably, in the present application, all undefined variables can be threshold values set by humans if not specifically defined.

[0036] This invention also provides an automated e-commerce data acquisition system, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the automated e-commerce data acquisition method. The automated e-commerce data acquisition system can run on computing devices such as desktop computers, laptops, handheld computers, and cloud data centers. The runnable system may include, but is not limited to, processors, memory, and server clusters. The processor executes the computer program within the following system units: The database construction unit is used to establish an e-commerce credit feature database with natural days as the collection granularity, which includes several credit features; The credit feature preprocessing unit is used to construct a stage coverage vector for each credit feature. The merchant comparison and screening unit is used to construct admission criteria to screen and compare merchants from the existing database; The offset recognition unit is used to construct a cold start offset recognition model based on the phase coverage vectors of the compared merchants and the merchants to be evaluated, and to obtain offset markers; An automated masking unit is used to mask e-commerce data collected during the period of trigger offset marker.

[0037] The beneficial effects of this invention are as follows: This invention provides an automated e-commerce data collection method and system. By characterizing the time point when the profile construction during the operation of new business opportunities exhibits abnormal comparison capabilities due to cold start drift, it can reasonably classify and filter merchant data at different operating stages and under different operating conditions in scenarios where merchant profiles are rapidly constructed using short-cycle, high-frequency data. This identifies and weakens the interference of cold start drift during the novice period on the profile, enabling a more balanced comparison between the short-term profile of the merchant to be evaluated and the historical profile of existing merchants in terms of operating stage and comparable indicators. Thus, without significantly lengthening the evaluation cycle, it improves the collection quality of profile construction data, thereby improving the accuracy and risk differentiation of platform merchant loan decisions and reducing the credit risk caused by profile mismatch when dealing with novice merchants. Attached Figure Description

[0038] The above and other features of the present invention will become more apparent from the detailed description of the embodiments shown in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals denote the same or similar elements. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort. In the drawings: Figure 1 The diagram shows a flowchart of an automated e-commerce data collection method. Figure 2An e-commerce data automatic collection system structure diagram is shown. DETAILED DESCRIPTION

[0039] The concept, specific structure and generated technical effects of the present application will be described clearly and completely in combination with embodiments and drawings to fully understand the purpose, scheme and effects of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0040] As Figure 1 An e-commerce data automatic collection method flowchart is shown. The e-commerce data automatic collection method according to the embodiment of the present application will be described below in combination with Figure 1 The method comprises the following steps: S100, establishing an e-commerce credit feature database with a natural day as a collection granularity, wherein the database comprises a plurality of credit features; S200, constructing a stage coverage vector for each credit feature; S300, constructing an access condition to screen and compare a merchant from a stock library; S400, constructing a cold start offset identification model according to the stage coverage vectors of the compared merchant and the merchant to be evaluated, and obtaining an offset label; S500, performing shielding on the e-commerce data collected in a period when the offset label is triggered.

[0041] Further, in step S100, the method of establishing an e-commerce credit feature database with a natural day as a collection granularity, wherein the database comprises a plurality of credit features, is as follows: the e-commerce credit feature database is used to store the values corresponding to each credit feature obtained by each merchant, a preset credit feature set is included therein, the credit feature set includes one or more of transaction size type features, transaction quality type features, operating efficiency type features and fund recycling type features; each natural day is taken as an observation point, and the value corresponding to the credit feature is obtained at each observation point.

[0042] The process of obtaining the credit feature at each observation point is as follows: for each merchant, the original operating data corresponding to each credit feature is extracted from the transaction system, the log system and the settlement system of the e-commerce platform in each natural day, so as to obtain the value of each credit feature in the natural day; and the values of each credit feature with the merchant identifier and the natural day as the index are written into the e-commerce credit feature database.

[0043] The transaction size type features include one or more of the total transaction amount, the number of payment orders, the number of independent buyers, or the average single customer price; the transaction quality type features include one or more of the number of returned orders, the return rate, the refund rate, and the proportion of dispute orders; the operating efficiency type features include one or more of the exposure amount, the number of visitors, the number of added-to-baskets, the conversion rate, and the added-to-basket rate; and the fund recovery type features include one or more of the amount of money received, the settlement amount, the withdrawal amount, and the final balance of the merchant account.

[0044] It should be noted that each feature in the credit feature set is stable and can be extracted from the internal system of the platform at a natural day granularity, and is closely related to the high-flow operating data of the repayment ability, so as to avoid mixing low-frequency static fields such as business information and category attributes into the credit feature set.

[0045] The values of each merchant for each credit feature corresponding to the natural day are organized into a credit feature record, and the merchant identifier and the natural day identifier are written into the e-commerce credit feature database as the primary key.

[0046] Further, in step S200, the method for constructing the phase coverage rate vector of each credit feature is: obtaining the observation duration of the to-be-evaluated merchant, intercepting each credit feature of each merchant within the current observation duration, any credit feature value sequence within the current observation duration is recorded as a feature observation sequence, the feature observation sequence is normalized, and the average value of each maximum value of the feature observation sequence is recorded as the peak value level; when an element in the feature observation sequence is greater than each element in the reverse time direction, the element is marked as a coverage peak, the coverage level is recorded as the proportion of the coverage peak in the feature observation sequence, and the product of the peak value level and the coverage level is recorded as the coverage rate of the credit feature. The coverage rate of each credit feature constitutes the phase coverage rate vector of the current natural day.

[0047] The observation duration Dur of the to-be-evaluated merchant refers to the number of natural days between the time point when the to-be-evaluated merchant starts to record the credit feature and the current time, which is constrained as Dur≥30 natural days; the current observation duration refers to each natural day in the Dur period in the reverse time direction from the current time.

[0048] Further, in step S300, the method for constructing the access condition to screen the comparison merchant from the inventory library is: the inventory library is a set of merchants in the e-commerce credit feature database, and the access conditions include: completing at least one merchant loan in the platform, having at least 90 days of valid operating records, and having the same or belonging to the same category cluster as the to-be-evaluated merchant; from the inventory library, the merchants that meet each access condition are taken as comparison merchants; The specific requirements for the merchant loan completed at least once in the platform include: the platform maintains a merchant loan record table for each merchant, in which the application time, approval result, credit limit, loan amount, repayment plan and repayment status information of each merchant loan are recorded. If the merchant has at least one record in the loan record table with a loan amount greater than zero and a loan state of having been loaned, and the state of the loan is normal repayment or has been cleared, it is determined that the merchant has completed at least one merchant loan in the platform.

[0049] Preferably, for records that are only in credit approval but the merchant has not actually used the loan, or records that have been rejected or canceled after application, they are not considered to have completed the merchant loan; for merchants who have been identified by the platform as maliciously overdue, fraudulent and listed in the blacklist, even if they have a loan record in history, they can not be included in the comparison of merchants in this embodiment, so that the merchants as a reference for comparison have normal performance in credit behavior.

[0050] Having at least 90 days of valid operation records means that within a 180-day observation period, the merchant counts the operation on each day, and the number of valid operation days exceeds 90 days to meet the condition, wherein when the natural day meets one of the following conditions, the natural day is recorded as the valid operation day of the merchant: a) There is a paid order in the natural day, and the number of transaction orders in the natural day is greater than zero; b) At least one transaction scale feature and one operating efficiency feature in the main credit features of the natural day have legal values, wherein the transaction scale feature is the total transaction amount or the number of paid orders, and the operating efficiency feature can be the number of visitors or the payment conversion rate; it is known that if there are abnormal records with missing time stamp, negative amount or significantly exceeding the preset value range in the corresponding credit feature record of the natural day, the natural day is not counted as a valid operation day.

[0051] The category range is the same as or belongs to the same category cluster of the merchant to be evaluated, which means that the e-commerce platform pre-sets the corresponding category code for each item for sale according to its own commodity classification system. The category code is divided according to the hierarchical structure. The platform calculates the contribution ratio of each category based on the transaction amount or the number of transaction orders of each category of each merchant within a preset observation period, and confirms one or several categories with the highest contribution ratio as the main category of the merchant.

[0052] Wherein the platform pre-aggregates multiple sub-categories into several category clusters according to business similarity, for example, multiple secondary categories such as men's clothing, women's clothing, and children's clothing can be attributed to a category cluster of the first-level category of clothing, multiple secondary categories such as mobile phones, tablets, and computers can be attributed to a category cluster of digital, and multiple secondary categories such as makeup and skin care can be attributed to a category cluster of the first-level category of beauty care. The category cluster to which the main category of each merchant belongs is the category cluster label defined by the merchant.

[0053] When screening the comparison merchants from the inventory library, the main category of the merchant to be evaluated and its corresponding category cluster label are obtained; among the candidate merchants, the merchants whose main category is the same as that of the merchant to be evaluated, or whose main category is not the same but belongs to the same category cluster, are selected as the merchants whose category range is the same as or belongs to the same category cluster as the merchant to be evaluated. Preferably, for merchants that operate multiple categories and have no obvious main category, the category cluster with the highest transaction amount proportion can be selected as the main category cluster of the merchant for comparison.

[0054] Further, in step S400, a cold start offset recognition model is constructed according to the phase coverage rate vectors of the comparison merchants and the merchant to be evaluated, and the method of obtaining the offset label is: taking the phase coverage rate vectors of different comparison merchants in the same natural day as a column, and the phase coverage rate vectors of different natural days of the same comparison merchant as a row to construct a matrix, which is recorded as the cold start offset recognition model; for any natural day, the Mahalanobis distance between the phase coverage rate vectors of the merchant to be evaluated and any comparison merchant is recorded as the cold start offset index, the median and the standard deviation of all cold start offset indexes are recorded as the cold start offset median and the offset fluctuation, respectively, and the quotient of the difference between the cold start offset index and the cold start offset median after the difference is calculated and the offset fluctuation is recorded as the benchmark offset index; for any comparison merchant, the ratio of the number of natural days with the benchmark offset index greater than 0 and less than 0 is recorded as the contrast coefficient, if the contrast coefficient is greater than or equal to 0.5, the comparison merchant is recorded as a high-contrast comparison merchant, otherwise it is recorded as a low-contrast comparison merchant; this step is based on the cold start offset recognition model to mark the offset.

[0055] For any given day, all baseline offset indices for high-contrast and low-contrast merchants are denoted as high-contrast offset vectors and low-contrast offset vectors, respectively. K-means clustering analysis is performed on the high-contrast and low-contrast offset vectors for all days. The cluster selection is implemented using a scree plot and the KMeans function from the scikit-learn library in Python. If the number of samples in a cluster is less than the average number of samples in a cluster, that cluster is designated as a collapsed cluster. The number of samples in a cluster refers to the number of elements in each cluster obtained from the clustering analysis, and the average number of samples in a cluster is the average of the corresponding number of samples in each cluster. The K-means clustering processes for the high-contrast and low-contrast offset vectors are independent of each other, therefore, the high-contrast and low-contrast offset vectors each constitute a collapsed cluster.

[0056] In the collapsed clusters of low-contrast offset vector clustering results, identify whether there are Lth consecutive natural days or more. If so, pre-offset labels are applied to them. For the natural days with pre-offset labels in the collapsed clusters of high-contrast offset vector clustering results, offset labels are applied.

[0057] Where Lth is the preset threshold for the number of natural days, with a default value of 3, a minimum value of 3, and an upper limit of 0.25 times the matrix time length.

[0058] Preferably, in step S400, a cold start offset identification model is constructed based on the phase coverage vectors of the comparison merchants and the merchant to be evaluated. The method for obtaining the offset marker is as follows: a matrix is ​​constructed with the phase coverage vectors of different comparison merchants on the same natural day as one row, and the phase coverage vectors of the same comparison merchant on different natural days as one column. This matrix is ​​denoted as the cold start offset identification model. Based on the coverage vector sequence over the entire time period, the dynamic time-normalized distance (DL) between the merchant to be evaluated and all comparison merchants is calculated for each natural day. The dynamic time-normalized distance is calculated using the distance function of the dtaidistance library in Python. The comparison merchants are divided into sk behavioral groups using the K-means algorithm, where sk is the number of clusters in the cluster, and its value ranges from sk∈[5,50]. The value of sk is adjusted according to the clustering effect. The average DL of each behavioral group is calculated, and the merchant to be evaluated is assigned to the behavioral group with the smallest average DL to form a temporal resonance cluster. The set of other comparison merchants in the temporal resonance cluster, excluding the merchant to be evaluated, is defined as the same reference group. DF is the modulus of the change amplitude of the coverage rate vector of each merchant in the time resonance cluster relative to the previous day, which is used to measure the overall change between the adjacent two natural days. For any merchant in the time resonance cluster, the difference between the current natural day's stage coverage rate vector and the previous day's stage coverage rate vector is calculated to obtain a coverage rate change vector. Then, the sum of the squares of the differences between the components of the coverage rate change vector is calculated, and the square root of the sum is obtained. The result is defined as the modulus DF of the coverage rate vector change amplitude of the natural day relative to the previous day. The average DF of all merchants on the same day is the DF average value. The natural day whose DF average value exceeds the upper quartile value is identified as a universal mutation point using a box plot. If the DF of the merchant to be evaluated exceeds the DF average value in the universal mutation point, it is recorded as an abnormally high frequency point, otherwise it is a potential high frequency point. If the DF of the merchant to be evaluated exceeds the DF average value in the non-universal mutation point, it is recorded as a potential abnormal point, otherwise it is an abnormally low frequency point. The center trend vector of the same type reference group on each natural day is calculated. The calculation method is as follows: the arithmetic mean of the vectors of all merchants in the same type reference group is calculated in each dimension, and the arithmetic mean is combined according to the original order (the order corresponding to the stage coverage rate vector) to form a new vector, which is the center trend vector of the natural day. The cosine similarity GcSim between the stage coverage rate vector of the merchant to be evaluated and the center trend vector is calculated, and a similarity time series is constructed. Any natural day and its previous Num natural days form a similarity observation window; Num is a preset integer variable, and its value range is Num∈[5,10]. The moving average GuEMA and the standard deviation GoErr of the similarity observation window are calculated. If the natural day satisfies GcSim≤GuEMA-2GoErr, it is determined to be a pattern deviation anomaly. The range of the coverage rate vector is defined as the stage difference Cs, and the sum of the coverage rates of each merchant in the same type reference group is defined as the total coverage Cv. The coverage rate consistency CI is calculated according to the total coverage: CI=|Cv-Cs| / ln(1+Cs+GcSim^2); if the natural day satisfies that the CI of the merchant to be evaluated is less than half of the average CI of all merchants, it is determined to be a dispersion anomaly; where ln() represents the logarithmic function with e as the base number. When the natural day is an abnormally high frequency point, it is marked as a deviation. When the natural day is an abnormally low frequency point, it is determined to be a non-anomaly point. When the natural day is a potential high frequency point and at least one of the pattern deviation anomaly or the dispersion anomaly is satisfied, it is marked as a deviation. When the natural day is a potential abnormal point and both the pattern deviation anomaly and the dispersion anomaly are satisfied, it is marked as a deviation.

[0059] Further, in step S500, the method of performing shielding on the e-commerce data collected in the offset marking period is: obtaining the offset marking set of the to-be-evaluated merchant in the step, marking the e-commerce data corresponding to the natural day in the offset marking set as shielding data, and ignoring the shielding data in subsequent construction of the credit portrait or calculation of the credit evaluation index for the to-be-evaluated merchant, and generating a credit evaluation result only based on the e-commerce data corresponding to the natural day not belonging to the offset marking time set.

[0060] For the to-be-evaluated merchant, the offset marking time set is used to record the natural day in which the merchant is determined to have obvious cold start offset by the cold start offset recognition model in the observation duration, and each element in the offset marking time set corresponds to a natural day or a continuous natural day interval.

[0061] In the prior art, the construction of the credit portrait or the calculation of the credit evaluation index is generally based on the historical business data of the merchant in a certain observation period, and after statistical summary according to the preset dimensions, the credit scoring model is input into the credit scoring model to complete. The e-commerce platform generally aggregates the original business data and repayment behavior data of the merchant in a fixed length time window according to the settlement period, calculates a group of statistical indexes such as daily average transaction amount, daily average order quantity, historical maximum transaction amount, index mean and variance, recent period growth rate, recent period return rate and overdue rate, takes these statistical indexes as the credit feature vector of the merchant, and inputs them into the pre-trained credit scoring model, usually adopts a logistic regression model or an integrated model based on a tree structure, to obtain the credit score, default probability estimate value and corresponding recommended credit limit and credit level of the merchant. The prior art generally regards the effective data in the entire observation period as samples of the same business stage, does not distinguish the business state of different time periods in detail, and does not have a data filtering mechanism for the cold start offset period. The present method is aimed at filtering data in such a cold start offset period.

[0062] In the e-commerce credit feature database, the merchant identifier and the natural day are used as joint indexes, the credit feature records corresponding to each natural day in the offset marking time set are located, and a shielding marking field is added to these records or the existing shielding marking field is updated to be in a shielding state. The shielding marking field is used to indicate whether the e-commerce data corresponding to the natural day is allowed to participate in the credit evaluation. When the shielding marking is in the shielding state, it indicates that the data of the natural day belongs to the cold start offset period and is ignored in the subsequent credit evaluation process. Preferably, the records in the shielding state do not require physical deletion of the original data, but are logically isolated through the shielding marking, so that the real business behavior in the offset period can still be traced back in subsequent audits or model iterations.

[0063] The e-commerce data automatic collection system provided by the embodiments of the present application, like Figure 2As shown is a structural diagram of an e-commerce data automatic collection system of the present application. The e-commerce data automatic collection system of this embodiment comprises a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the e-commerce data automatic collection method embodiment described above are implemented.

[0064] The system comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to run in the units of the following system: A database construction unit for constructing an e-commerce credit feature database with a natural day as the collection granularity, which includes a plurality of credit features. A credit feature preprocessing unit for constructing a stage coverage vector for each credit feature. A comparison merchant screening unit for constructing an access condition to screen comparison merchants from the inventory library. An offset identification unit for constructing a cold start offset identification model according to the stage coverage vectors of the comparison merchants and the merchants to be evaluated to obtain an offset label. An automatic shielding unit for shielding the e-commerce data collected during the period when the offset label is triggered.

[0065] The e-commerce data automatic collection system can run in desktop computers, notebook computers, palmtop computers, and cloud servers, etc. The system that the e-commerce data automatic collection system can run in can include, but is not limited to, a processor and a memory. Those skilled in the art can understand that the examples are only examples of the e-commerce data automatic collection system and do not constitute a limitation on the e-commerce data automatic collection system. It can include more or fewer components, or combine certain components, or different components, for example, the e-commerce data automatic collection system can also include input and output devices, network access devices, buses, etc.

[0066] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is a control center of the e-commerce data automatic acquisition system running system, and connects each part of the e-commerce data automatic acquisition system running system through various interfaces and lines.

[0067] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the e-commerce data automatic acquisition system by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc. The data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0068] Although the description of the present application has been quite detailed and particularly described with respect to several embodiments, it is not intended to be limited to any of these details or embodiments or any special embodiment, so as to effectively cover the intended scope of the present application. In addition, the present application is described above in the embodiments that the inventors can foresee, and the purpose is to provide a useful description, and those non-essential modifications to the present application that have not yet been foreseen can still represent equivalent modifications of the present application.

Claims

1. An e-commerce data automatic collection method, characterized in that, The method comprises the following steps: S100, establishing an e-commerce credit feature database with a natural day as a collection granularity, wherein the database comprises a plurality of credit features; S200, constructing a stage coverage vector for each credit feature; S300, constructing an access condition to screen comparison merchants from a stock library; S400, constructing a cold start offset identification model according to the stage coverage vectors of the comparison merchants and the merchant to be evaluated, and obtaining an offset label; S500, performing shielding on e-commerce data collected in a period when the offset label is triggered; The method of S400 is to calculate a cold start offset index based on Mahalanobis distance, obtain a cold start offset median and an offset fluctuation amount by using a median and a standard deviation, and then form a benchmark offset index and a comparison coefficient to distinguish high-contrast comparison merchants and low-contrast comparison merchants; then perform Kmeans clustering analysis on the high-contrast offset vector and the low-contrast offset vector respectively, identify collapsed clusters according to the number of samples and the average number of samples of the clusters, identify pre-offset labels with a time sequence of a plurality of natural days in the collapsed clusters of the low-contrast offset vector, and perform offset labeling on the natural days with the pre-offset labels in the collapsed clusters of the high-contrast offset vector.

2. The method of claim 1, wherein, In step S100, the method of establishing an e-commerce credit feature database with a natural day as a collection granularity, wherein the database comprises a plurality of credit features, is as follows: the e-commerce credit feature database is used to store the corresponding values of each credit feature obtained by each merchant, a preset credit feature set is included in the database, the credit feature set includes one or more of transaction size features, transaction quality features, operating efficiency features and fund recycling features; each natural day is taken as an observation point, and the corresponding value of the credit feature is obtained at each observation point.

3. The method of claim 1, wherein, In step S200, the method of constructing a stage coverage vector for each credit feature is as follows: the observation length of the merchant to be evaluated is obtained, each credit feature in the current observation length is intercepted for each merchant, a sequence formed by each value of any credit feature in the current observation length is recorded as a feature observation sequence, the feature observation sequence is normalized, the average value of each maximum value of the feature observation sequence is recorded as a peak value level; when an element in the feature observation sequence is greater than each element in the reverse time direction, the element is marked as a coverage peak value, the proportion of the coverage peak value in the feature observation sequence is recorded as a coverage level, and the product of the peak value level and the coverage level is recorded as the coverage rate of the credit feature; and the stage coverage vector is constructed by the coverage rates of each credit feature.

4. The method of claim 1, wherein, In step S300, the method of constructing an access condition to screen comparison merchants from a stock library is as follows: the stock library is a set of each merchant in the e-commerce credit feature database, and the access conditions include: completing at least one merchant loan in the platform, having at least 90 days of valid operation records, and having the same category range or belonging to the same category cluster as the merchant to be evaluated; the merchants satisfying each access condition are taken as comparison merchants from the stock library.

5. The method of claim 1, wherein, In step S400, a cold start offset identification model is constructed according to the phase coverage rate vectors of the comparison merchants and the merchant to be evaluated, and the method for obtaining the offset label is: the phase coverage rate vectors of different comparison merchants in the same natural day are taken as a column, and the phase coverage rate vectors of different natural days of the same comparison merchant are taken as a row to construct a matrix, which is recorded as a cold start offset identification model; for any natural day, the Mahalanobis distance of the phase coverage rate vectors of the comparison merchant and the merchant to be evaluated is recorded as a cold start offset index, the median and the standard deviation of all cold start offset indexes are recorded as a cold start offset median and an offset fluctuation, respectively, and the quotient of the difference between the cold start offset index and the cold start offset median after the difference is calculated and the offset fluctuation is recorded as a benchmark offset index; For any comparison merchant, the ratio of the number of natural days with the benchmark offset index greater than 0 and less than 0 is recorded as a contrast coefficient, if the contrast coefficient is greater than or equal to 0.5, the comparison merchant is recorded as a high-contrast comparison merchant, otherwise, the comparison merchant is recorded as a low-contrast comparison merchant; For any natural day, all benchmark offset indexes of high-contrast comparison merchants and low-contrast comparison merchants are recorded as high-contrast offset vectors and low-contrast offset vectors, respectively; Kmeans clustering analysis is performed on the high-contrast offset vectors and the low-contrast offset vectors of all natural days, and if the number of samples in a cluster is less than the average cluster sample number, the cluster is recorded as a collapsed cluster; In the collapsed cluster of the low-contrast offset vector clustering result, whether there is a time-sequentially continuous Lth natural day or more is identified, and if there is, a pre-offset label is marked; the natural days with the pre-offset label in the collapsed cluster of the high-contrast offset vector clustering result are marked with an offset label.

6. The method of claim 1, wherein, In step S400, a cold start offset identification model is constructed according to the phase coverage rate vectors of the comparison merchants and the merchant to be evaluated, and the method for obtaining the offset label is: the phase coverage rate vectors of different comparison merchants in the same natural day are taken as a row, and the phase coverage rate vectors of different natural days of the same comparison merchant are taken as a column to construct a matrix, which is recorded as a cold start offset identification model: Based on the coverage rate vector sequence of the whole time period, the dynamic time warping distance DL of each natural day between the merchant to be evaluated and all comparison merchants is calculated, the comparison merchants are divided into sk behavior groups by using the Kmeans algorithm, wherein sk is the number of clusters in the clustering, the average value of DL of each behavior group is calculated, the merchant to be evaluated is classified into the behavior group with the smallest average value of DL to form a time-sequentially resonant cluster, and the set of other comparison merchants in the time-sequentially resonant cluster except the merchant to be evaluated is defined as a same-type reference group; In each natural day, the modulus of the change amplitude of the coverage rate vector relative to the previous natural day is calculated and recorded as DF; the average value of DF of all merchants on the same day is DF average value, and the natural day with the DF average value exceeding the upper quartile value is identified by using a box plot and recorded as a universal mutation point; If in the universal mutation point, if the DF of the merchant to be evaluated is higher than the DF average, it is recorded as an abnormal high frequency point, otherwise it is a potential high frequency point; if in the non-universal mutation point, if the DF of the merchant to be evaluated is higher than the DF average, it is recorded as a potential abnormal point ； The cosine similarity GcSim between the stage coverage vector of the merchant to be evaluated and the central tendency vector of the same type reference group on each natural day is calculated, and a similarity time series is constructed. Any natural day and its previous Num natural days form a similarity observation window. The moving average GuEMA and the standard deviation GoErr of the similarity observation window are calculated. If the natural day satisfies GcSim≤GuEMA-2GoErr, it is determined to be a mode deviation anomaly; The range of the coverage vector is defined as the stage difference Cs, and the sum of the coverage rates of each merchant in the same type reference group is recorded as the total coverage Cv. The coverage consistency CI is calculated according to the total coverage: CI=|Cv-Cs| / ln(1+Cs+GcSim^2). If the natural day satisfies that the CI of the merchant to be evaluated is less than half of the average CI of all merchants, it is determined to be a dispersion anomaly. When the natural day is an abnormally high frequency point, a shift flag is marked. When the natural day is a potential high frequency point and at least one of the mode deviation anomaly or the dispersion anomaly is satisfied, a shift flag is marked. When the natural day is a potential anomaly point and both the mode deviation anomaly and the dispersion anomaly are satisfied, a shift flag is marked.

7. The method of claim 1, wherein, In step S500, the method of shielding the e-commerce data collected in the period of triggering the shift flag is: obtaining the shift flag set of the merchant to be evaluated in step S500, marking the e-commerce data corresponding to the natural day in the shift flag set as shielding data, and ignoring the shielding data when constructing the credit portrait or calculating the credit evaluation index for the merchant to be evaluated in the subsequent steps. Only based on the e-commerce data corresponding to the natural day not belonging to the shift flag time set, the credit evaluation result is generated.

8. An e-commerce data automated collection system characterized in that, The e-commerce data automatic collection system includes a processor, a memory, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the steps in the e-commerce data automatic collection method of any one of claims 1-7. The e-commerce data automatic collection system runs in a computing device of a desktop computer, a notebook computer, a palm computer, and a cloud data center.

Images