Cross-border e-commerce big data anomaly detection method and system

Abstract

The invention provides a cross-border e-commerce big data anomaly detection method and system, and relates to the technical field of data processing, and the method comprises the steps: obtaining historical cross-border e-commerce transaction data; determining a feature dictionary for describing normal transaction behaviors of the historical cross-border e-commerce transaction data and an original reconstruction error of the historical cross-border e-commerce transaction data; obtaining current cross-border e-commerce transaction data; performing data reconstruction on the current cross-border e-commerce transaction data through the feature dictionary, and determining a current reconstruction error of the current cross-border e-commerce transaction data; when the current reconstruction error is greater than the original reconstruction error, determining whether the current cross-border e-commerce transaction data is abnormal; calculating a contribution value of each transaction feature to a current reconstruction error through a Shapley value; and outputting the transaction characteristics corresponding to the maximum contribution value as abnormal cross-border e-commerce transaction data. The accuracy of anomaly detection is effectively improved, the operation smoothness of the e-commerce platform caused by frequent misinformation is avoided, and the maintenance cost is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method and system for detecting anomalies in cross-border e-commerce big data. Background Art

[0002] Cross-border e-commerce big data refers to a multi-dimensional, diversified set of structured and unstructured data generated in cross-border e-commerce activities, covering transactions, users, logistics, operations and other links. These data sources include order transaction records (such as order amount, product classification, payment method), user behavior (such as browsing path, click frequency, length of stay), platform operation data (such as visit volume, transaction rate, bounce rate) and external factors (such as exchange rate, tariff, logistics information, etc.). Its characteristics are large scale, real-time dynamics, high complexity, and because it involves multi-national and multi-regional businesses, it has significant cross-border characteristics, such as the complexity of multiple languages, multiple currencies and multiple time zones.

[0003] Cross-border e-commerce big data anomaly detection is a key link in ensuring platform security and operational efficiency. By detecting abnormal data, abnormal transaction behaviors (such as malicious order fraud, bulk returns or fraudulent orders) and platform operation problems (such as abnormal bounce rates or visit volume fluctuations) can be identified in a timely manner to avoid platform profit losses, reduced user experience and increased business risks. In addition, big data anomaly detection can also optimize platform operation strategies, improve data utilization efficiency, and ensure the long-term sustainable development of cross-border e-commerce.

[0004] However, the existing cross-border e-commerce big data anomaly detection process often directly measures whether the data is abnormal based on fixed transaction data values, which cannot adapt to the complex and changeable cross-border e-commerce scenarios, and easily leads to frequent false alarms and missed alarms, increasing the maintenance cost of the cross-border e-commerce platform and reducing the operation smoothness of the cross-border e-commerce platform. Summary of the invention

[0005] In order to solve the technical problem that the existing cross-border e-commerce big data anomaly detection process in the prior art often directly measures whether the data is abnormal based on fixed transaction data values, cannot adapt to complex and changeable cross-border e-commerce scenarios, easily leads to frequent false alarms and missed alarms, increases the maintenance cost of the cross-border e-commerce platform, and reduces the running smoothness of the cross-border e-commerce platform, the present invention provides a cross-border e-commerce big data anomaly detection method and system.

[0006] The technical solution provided by the embodiment of the present invention is as follows:

[0007] First aspect

[0008] An embodiment of the present invention provides a cross-border e-commerce big data anomaly detection method, comprising:

[0009] S1: Obtain historical cross-border e-commerce transaction data including different transaction characteristics within a preset period;

[0010] S2: Reconstruct the historical cross-border e-commerce transaction data based on dictionary learning with the goal of minimizing the reconstruction error of the historical cross-border e-commerce transaction data, and combine the simulated annealing algorithm to determine the feature dictionary that describes the normal transaction behavior of the historical cross-border e-commerce transaction data and the original reconstruction error of the historical cross-border e-commerce transaction data;

[0011] S3: Get current cross-border e-commerce transaction data;

[0012] S4: reconstruct the current cross-border e-commerce transaction data through the feature dictionary to determine the current reconstruction error of the current cross-border e-commerce transaction data;

[0013] S5: If the current reconstruction error is greater than the original reconstruction error, it is determined that the current cross-border e-commerce transaction data is abnormal, and the process goes to step S6; otherwise, the process returns to step S3;

[0014] S6: Calculate the contribution of each transaction feature to the current reconstruction error through the Shapley value;

[0015] S7: Output the transaction feature corresponding to the maximum contribution value as abnormal cross-border e-commerce transaction data.

[0016] Second aspect

[0017] An embodiment of the present invention provides a cross-border e-commerce big data anomaly detection system, comprising:

[0018] processor;

[0019] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the cross-border e-commerce big data anomaly detection method as described in the first aspect is implemented.

[0020] The third aspect

[0021] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, the cross-border e-commerce big data anomaly detection method as described in the first aspect is implemented.

[0022] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0023] In the present invention, by performing dictionary learning on the historical cross-border e-commerce transaction data acquired within a preset period, the feature dictionary describing the normal transaction behavior of the historical cross-border e-commerce transaction data and the original reconstruction error of the historical cross-border e-commerce transaction data are determined in combination with the simulated annealing algorithm, and then the feature dictionary learned by the dictionary is used to reconstruct the current cross-border e-commerce transaction data, and the abnormal data is determined by taking the original reconstruction error as the judgment basis of the abnormal data, effectively capturing the normal mode of the historical transaction data, avoiding falling into the local optimum, and generating a more globally representative feature dictionary. Combined with the feature dictionary, the original reconstruction error is used as the benchmark to detect abnormal data, ensuring that the abnormal detection is not disturbed by local data, with high accuracy and robustness, and can adapt to complex transaction scene changes, improve the accuracy of detection and the adaptability to dynamic anomalies. After determining the abnormal data, the contribution value of each transaction feature to the current reconstruction error is calculated by the Shapley value, and the transaction feature with the largest collinearity of the current reconstruction error of the abnormal data is determined as the output of the abnormal cross-border e-commerce transaction data, accurately quantifying the impact of each feature on the anomaly, and locating the key abnormal features. By accurately identifying the features that have the greatest impact on anomalies and precisely locating the source of anomalies, a clear basis can be provided for rapid diagnosis and targeted optimization of abnormal problems. This can effectively reduce the frequency of false alarms on cross-border e-commerce platforms and greatly reduce maintenance costs and operating efficiency of cross-border e-commerce platforms. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A schematic diagram of a process for detecting anomalies in cross-border e-commerce big data provided by an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of the structure of a cross-border e-commerce big data anomaly detection system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0028] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "example" is intended to present the concept in a specific way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0029] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0030] Reference Manual Attached Figure 1 , showing a flow chart of a cross-border e-commerce big data anomaly detection method provided by an embodiment of the present invention.

[0031] The embodiment of the present invention provides a cross-border e-commerce big data anomaly detection method, which can be implemented by a cross-border e-commerce big data anomaly detection device, which can be a terminal or a server. The processing flow of the cross-border e-commerce big data anomaly detection method may include the following steps:

[0032] S1: Obtain historical cross-border e-commerce transaction data including different transaction characteristics within a preset period.

[0033] Among them, the preset period refers to the time period determined based on historical data analysis, which is used to divide the periodic characteristics of cross-border e-commerce transaction data. This can be divided by day, week, month, quarter or year. Historical cross-border e-commerce transaction data refers to the multi-dimensional data accumulated by the cross-border e-commerce platform within the preset period. By obtaining historical cross-border e-commerce transaction data within the preset period, the characteristics and rules of transactions and user behaviors are captured, providing basic data support for subsequent dictionary learning and anomaly detection. This process helps to clearly define normal transaction behavior patterns and ensure the accuracy and robustness of anomaly detection.

[0034] In a possible implementation, the transaction characteristics include order amount, payment time interval, order item quantity, return quantity, cross-border e-commerce platform visit volume, cross-border e-commerce platform bounce rate, and cross-border e-commerce platform order transaction volume.

[0035] Transaction characteristics include order-related data and platform operation data, among which order-related data include order amount: the total amount of a single order, which is usually used to monitor abnormal large orders. Payment time interval: the time from when the user places an order to when the user pays, reflecting the user's behavior pattern. Order item quantity: the number of items included in the order, used to detect abnormal bulk purchases. Return volume: the number of returns for a single user or a certain time period, to monitor abnormal return behavior. Platform operation data includes traffic data: the number of visits to the entire platform in a certain time period. Bounce rate: the proportion of people leaving immediately after visiting the homepage or a certain page, reflecting whether the user behavior is abnormal. Order volume: the changing trend of the overall number of orders on the platform, used to discover whether there are abnormal fluctuations on the platform.

[0036] In a possible implementation manner, the method for determining the preset period specifically includes:

[0037] S101: Obtain historical cross-border e-commerce transaction data of the cross-border e-commerce platform.

[0038] S102: Standardize historical cross-border e-commerce transaction data:

[0039]

[0040] Among them, x(t) represents the historical cross-border e-commerce transaction data at time t, x′(t) represents the standardized x(t), μ and σ represent the mean and variance of x(t), respectively.

[0041] S103: Perform Fourier transform on the historical cross-border e-commerce transaction data to obtain the frequency domain characteristics of the historical cross-border e-commerce transaction data:

[0042] X′(f)=FFT(x′(t))

[0043] Wherein, FFT represents fast Fourier transform, X′(f) represents the energy distribution of x′(t) at different frequencies f, that is, the frequency domain signal obtained after x′(t) is transformed by FFT.

[0044] It should be noted that the use of Fourier transform to extract frequency domain features and combine spectral entropy to analyze the periodicity of trading data can not only accurately identify the main cycle, but also avoid the influence of noise or random interference through entropy constraints. At the same time, this method can effectively deal with complex data scenarios with multiple cycles or high noise, improve the robustness and business applicability of cycle selection, and significantly improve the accuracy and reliability of anomaly detection.

[0045] S104: Determine a preset period in the frequency domain features in combination with the spectrum entropy value of the frequency domain signal:

[0046]

[0047] Among them, Tmain Indicates the preset period, f main represents the main frequency of the frequency domain feature, Re(X'(f)) and Im(X'(f)) represent the real part and imaginary part of the frequency domain signal respectively, arg max f |X'(f)| represents the value f that maximizes the frequency domain signal amplitude |X'(f)|, log represents the logarithmic function, P(f) represents the probability distribution of f, H f Represents the spectral entropy value about f.

[0048] It should be noted that by introducing spectral entropy and frequency energy distribution, not only the main frequency corresponding to the maximum amplitude is considered, but also the entropy constraint is introduced to measure the complexity of the frequency distribution. By introducing spectral entropy, the complexity and periodicity of the signal frequency distribution are quantified. A low spectral entropy value means that the frequency distribution is more concentrated and the signal periodicity is stronger. Therefore, when selecting the main frequency, combining the amplitude and low entropy constraints can effectively avoid the wrong main frequency introduced by noise or random interference, and improve the reliability of period selection. Compared with the traditional method that relies solely on amplitude, spectral entropy provides a global measure of period significance and signal regularity, making this method particularly suitable for processing complex trading data with multi-frequency components or high noise, further improving the accuracy and business applicability of period calculation. Therefore, frequencies with more significant periodicity and less noise interference are preferred. Compared with the traditional method of directly selecting the maximum amplitude, this method is more suitable for processing trading data with multiple periods or random noise. Combined with probability normalization and weight optimization, the key periodic characteristics in the signal can be fully captured, significantly improving the accuracy and robustness of the preset period.

[0049] S2: Reconstruct the historical cross-border e-commerce transaction data based on dictionary learning with the goal of minimizing the reconstruction error of the historical cross-border e-commerce transaction data, and combine the simulated annealing algorithm to determine the feature dictionary that describes the normal transaction behavior of the historical cross-border e-commerce transaction data and the original reconstruction error of the historical cross-border e-commerce transaction data.

[0050] Among them, dictionary learning is a machine learning method that learns a set of sparse features (dictionary D) from historical data to efficiently represent the original data X. The goal is to find a dictionary D so that X can be represented by a sparse coefficient α. Reconstruction error refers to the difference between the original data and the data reconstructed by the model (dictionary learning). Data reconstruction refers to the approximate restoration of the original data using the learned dictionary D and the sparse representation α. ​​The reconstruction accuracy depends on the quality of the dictionary and the representation ability of the sparse coefficient. Simulated annealing is a global optimization algorithm that simulates the state changes of the physical system during the annealing process, searches for the optimal solution by gradually lowering the "temperature" to avoid falling into the local optimum. Its core is to dynamically control the search range so that the early exploration is strong and the later stage tends to be stable. The feature dictionary is a set of feature templates extracted from historical data through dictionary learning, which is used to represent normal trading behavior. Each column of the dictionary is a feature vector that can be used to effectively reconstruct the original data.

[0051] It should be noted that dictionary learning is used to minimize the reconstruction error of historical cross-border e-commerce transaction data and extract a feature dictionary that describes normal transaction behavior. At the same time, the dictionary is optimized by combining the simulated annealing algorithm to avoid falling into the local optimal solution, ensuring that the feature dictionary can better represent the global transaction rules, thereby calculating the optimal reconstruction error of the original data. This process provides a high-precision benchmark model and judgment criteria for subsequent anomaly detection, which helps to improve the accuracy of anomaly detection.

[0052] In a possible implementation, S2 specifically includes:

[0053] S201: Initialize a feature dictionary and a sparse coefficient matrix with the goal of describing historical cross-border e-commerce transaction data, wherein the sparse coefficient matrix is ​​used to describe the sparse representation of historical cross-border e-commerce transaction data in the feature dictionary:

[0054] X≈Dα

[0055] X=[x1,x2,…,x n ]

[0056] D=[d1,d2,…,d k ]

[0057] α=[α1,α2,…,α n ]

[0058] Among them, X represents the matrix composed of historical cross-border e-commerce transaction data, D represents the feature dictionary, α represents the sparse coefficient matrix, and x i represents the i-th sample data in the historical cross-border e-commerce transaction data, i = 1, 2, ..., n, n represents the total number of sample data, d j The feature vector representing the jth transaction feature in the historical cross-border e-commerce transaction data, j = 1, 2, ..., k, k represents the total number of transaction features, αi Represents x i Sparse representation on D.

[0059] Optionally, the PCA principal components of the historical cross-border e-commerce transaction data are extracted, and the feature dictionary is initialized based on the principal components. The sparse matrix can be initialized to a zero matrix or a random matrix, and the sparse coding method OMP can be used for calculation during the update process, that is, α = OMP (D, X).

[0060] S202: Taking minimizing the reconstruction error of historical cross-border e-commerce transaction data as the update termination target, and using the update termination target as the energy function of the simulated annealing algorithm to update the feature dictionary and the sparse coefficient matrix, the update termination target is specifically:

[0061]

[0062] Among them, min means taking the minimum value, represents the square of the Frobenius norm, || ||1 represents the L1 norm, and λ represents the sparsity weight coefficient.

[0063] In a possible implementation, S202 specifically includes:

[0064] S2021: Initialize annealing temperature T and energy function value:

[0065]

[0066] Where E represents the energy function value with feature dictionary D and sparse coefficient matrix α.

[0067] S2022: Add random perturbations to feature dictionaries and sparse coefficient matrices:

[0068] D′=D+ΔD

[0069] α′=α+Δα

[0070] Among them, ΔD and Δα represent random feature dictionary perturbations and random sparse coefficient matrix perturbations, respectively, and D′ and α′ represent the feature dictionary and sparse coefficient matrix after adding random perturbations, respectively.

[0071] S2023: Calculate the energy function value after adding random perturbations:

[0072]

[0073] Among them, E′ represents the energy function value after adding random perturbations.

[0074] S2024: The energy function difference between the two energy function values ​​calculated:

[0075] ΔE=E′-E

[0076] Where ΔE represents the energy function difference.

[0077] S2025: When the absolute value of the energy function difference is less than the preset absolute value, proceed to step S2028; otherwise, proceed to step S2026.

[0078] It should be noted that those skilled in the art can set the size of the preset absolute value according to actual needs, and the present invention is not limited here.

[0079] S2026: Update the feature dictionary and sparse coefficient matrix according to the energy function difference:

[0080]

[0081] r t+1 =μr t (1-r t ),μ∈[3.5,4]

[0082] Where exp represents the natural exponential function, μ represents the control parameter that controls the chaotic characteristics of random numbers, and r t and r t+1 They represent the random numbers of the tth update process and the t+1th update process respectively.

[0083] It should be noted that the use of chaotic systems to generate random number sequences enhances the complexity of pseudo-random numbers, which helps to improve the global search capability and adaptability of the simulated annealing algorithm. Moreover, while providing pseudo-randomness, it avoids the uncontrollability of complete randomness.

[0084] S2027: Lower the annealing temperature and return to step S2022:

[0085] T t+1 =T t γ t

[0086]

[0087] Among them, T t and T t+1 They represent the annealing temperatures of the tth update and the t+1th update process, γ0 represents the initial temperature attenuation coefficient, and γ t represents the temperature attenuation coefficient at the tth update, and log represents the logarithmic function.

[0088] It should be noted that the temperature attenuation coefficient is adjusted by a logarithmic function to make the cooling process smoother, especially in the early stage of optimization, the temperature drops slowly, enhancing the global exploration ability, while in the later stage, the temperature decay speed is accelerated to promote rapid convergence. At the same time, this update method is based on a simple mathematical relationship and is easy to implement and control.

[0089] S2028: Output the current feature dictionary and the current sparse coefficient matrix as optimization results.

[0090] It should be noted that the combination of random perturbations and dynamic temperature adjustments can help improve the global search capability by avoiding falling into the local optimal solution. By calculating the difference in energy function values, the optimization path can be reasonably screened to enhance the stability and robustness of the model. In addition, the pseudo-random number sequence generated by the chaotic system increases the search diversity and improves the adaptability of the search results. The use of logarithmic functions to adjust the annealing temperature attenuation coefficient makes the early exploration more extensive and the later convergence faster, which significantly improves the optimization efficiency and the accuracy of the results. This method is particularly suitable for nonlinear optimization of complex data, and can accurately obtain better feature dictionaries and sparse coefficient matrices, laying a solid foundation for subsequent anomaly detection.

[0091] S203: Calculate the original reconstruction error based on the updated feature dictionary and sparse coefficient matrix:

[0092]

[0093] Among them, e baseline represents the original reconstruction error, and || ||2 represents the L2 norm.

[0094] It should be noted that by combining sparse dictionary learning and simulated annealing algorithms, the feature dictionary and sparse coefficient matrix can be optimized globally, significantly reducing the reconstruction error of historical cross-border e-commerce transaction data. Sparse dictionary learning captures the potential sparse characteristics in transaction data, making the representation of data in the dictionary more compact and efficient. The introduction of simulated annealing algorithm can effectively avoid the optimization from falling into local optimality, ensuring that the search results for feature dictionary and sparse coefficient matrix are more accurate. In addition, calculating the reconstruction error and using it as the optimization target provides a quantifiable judgment benchmark, thereby ensuring that the optimization process is more reliable and explainable, laying a precise foundation for subsequent anomaly detection.

[0095] S3: Get current cross-border e-commerce transaction data.

[0096] Among them, current cross-border e-commerce transaction data refers to the latest transaction-related data generated by the cross-border e-commerce platform in a real-time or near real-time environment, including orders, user behavior, platform operation and other dimensions.

[0097] S4: Reconstruct the current cross-border e-commerce transaction data through the feature dictionary to determine the current reconstruction error of the current cross-border e-commerce transaction data.

[0098] In a possible implementation manner, the current reconstruction error is calculated as follows:

[0099]

[0100] Among them, X current represents the matrix composed of current cross-border e-commerce transaction data, α current Reconstruct X using D current The sparse coefficient matrix, e current Represents the current reconstruction error.

[0101] It should be noted that by reconstructing the current transaction data using the feature dictionary and sparse coefficient matrix and calculating the current reconstruction error, the degree of deviation between the data and the model can be accurately quantified. This method is based on the sparse representation theory and can efficiently compress and restore transaction data. At the same time, by optimizing the sparsity weight coefficient, it controls the sparsity and error balance of data reconstruction. This approach not only takes into account the universality of the global feature dictionary, but also adapts to the specific characteristics of the current data, thereby significantly improving the accuracy of reconstruction and the sensitivity of anomaly detection. Especially in the case of high data complexity or high noise, this method can more stably capture the key change characteristics of the data and improve the accuracy and robustness of anomaly detection.

[0102] S5: When the current reconstruction error is greater than the original reconstruction error, it is determined that there is an abnormality in the current cross-border e-commerce transaction data, and the process goes to step S6; otherwise, the process returns to step S3.

[0103] S6: Calculate the contribution of each transaction feature to the current reconstruction error through the Shapley value.

[0104] Among them, the Shapley value originates from game theory and is used to measure the marginal contribution of each participant (feature or variable) to the overall result. The contribution value refers to the specific degree of influence of the feature on the reconstruction error. The Shapley value calculates the relative contribution of each feature to the current reconstruction error through the marginal contribution. The higher the contribution value, the more significant the impact of the feature on the current anomaly. The marginal contribution of each transaction feature to the current reconstruction error is calculated through the Shapley value, and the importance of each feature is quantified, so as to identify the features that have the greatest impact on the abnormal data. This method can not only effectively locate the source of the anomaly, but also improve the interpretability and accuracy of anomaly detection.

[0105] In a possible implementation, the contribution value is calculated as follows:

[0106]

[0107] Among them, φ i represents the Shapley value of the i-th transaction feature to the current reconstruction error, i.e., the contribution value. N represents the set of all transaction features. {i} represents any transaction feature set that does not include transaction feature i, |S| represents the number of transaction features in transaction feature set S, |N| represents the number of transaction features in transaction feature set N, ! represents cumulative multiplication, E(S) represents the current reconstruction error after sparse reconstruction using S, and E(S∪{i}) represents the current reconstruction error recalculated after adding feature i to feature subset S.

[0108] It should be noted that by using the Shapley value to calculate the contribution value, the marginal contribution of each transaction feature to the current reconstruction error can be accurately measured. This method not only considers the individual impact of the feature, but also quantifies the synergy between features by combining and analyzing all possible feature subsets, thereby providing a more comprehensive and fair assessment of feature importance. At the same time, the game-theoretic basis of the Shapley value ensures the interpretability and robustness of the calculation results. By comparing the changes in the reconstruction error before and after adding specific features, this method can effectively locate the most critical abnormal features and significantly improve the accuracy of anomaly detection. This calculation method is particularly effective when there are complex interactions or redundancies between features.

[0109] S7: Output the transaction feature corresponding to the maximum contribution value as abnormal cross-border e-commerce transaction data.

[0110] In a possible implementation manner, after S7, the method further includes:

[0111] Issue early warning information about abnormal cross-border e-commerce transaction data.

[0112] It is understandable that after detecting abnormal cross-border e-commerce transaction data, timely issuing early warning information can help the platform quickly respond to potential risks, such as large fraudulent orders, bulk malicious purchases, etc., to prevent possible economic losses or operational abnormalities. This mechanism can improve the security and real-time performance of the system and help the platform's efficient management and risk control.

[0113] In the actual application process, first, by obtaining historical transaction data within a preset period, the rules of transaction and user behavior are extracted. Then, based on dictionary learning and simulated annealing algorithms, the historical data is optimized and modeled to generate a feature dictionary describing normal transaction behavior, and the goal is to minimize the reconstruction error, providing a reliable benchmark for subsequent detection. Then the current transaction data is obtained, reconstructed using the feature dictionary, the current reconstruction error is calculated, and the abnormal data is determined by comparing it with the original reconstruction error. Further, the contribution of the feature to the reconstruction error is calculated by the Shapley value, the feature with the greatest impact on the anomaly is identified, and finally the abnormal transaction feature is output. By obtaining historical cross-border e-commerce transaction data within a preset period, the feature dictionary is extracted using dictionary learning and simulated annealing algorithms, and modeling is performed with the goal of minimizing the reconstruction error, providing an accurate benchmark for anomaly detection. Then, the current transaction data is reconstructed using the feature dictionary, the current reconstruction error is calculated, and the presence of anomalies is determined by comparing the original error. The contribution of each feature to the error is evaluated using the Shapley value, and the feature with the greatest impact on the anomaly is accurately identified. This method combines global optimization, sparsity modeling and feature contribution analysis, which can not only effectively avoid local optimality, but also improve the robustness and interpretability of anomaly detection, ensure the accuracy and business applicability of the results, and thus enhance the real-time risk management capabilities and security of cross-border e-commerce platforms.

[0114] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0115] In the present invention, by learning the dictionary of the historical cross-border e-commerce transaction data acquired within the preset period, the feature dictionary describing the normal transaction behavior of the historical cross-border e-commerce transaction data and the original reconstruction error of the historical cross-border e-commerce transaction data are determined in combination with the simulated annealing algorithm, and then the feature dictionary learned by the dictionary is used to reconstruct the current cross-border e-commerce transaction data, and the abnormal data is determined by taking the original reconstruction error as the judgment basis of the abnormal data, effectively capturing the normal mode of the historical transaction data, avoiding falling into the local optimum, and generating a more globally representative feature dictionary. Combined with the feature dictionary, the abnormal data is detected based on the original reconstruction error, ensuring that the abnormal detection is not disturbed by the local data, with high accuracy and robustness, and can adapt to the changes in complex transaction scenarios, and improve the accuracy of detection and the adaptability to dynamic anomalies. After determining the abnormal data, the contribution value of each transaction feature to the current reconstruction error is calculated by the Shapley value, and the transaction feature with the largest collinearity of the current reconstruction error of the abnormal data is determined as the output of the abnormal cross-border e-commerce transaction data, accurately quantifying the influence of each feature on the anomaly, and locating the key abnormal features. By accurately identifying the features that have the greatest impact on anomalies and precisely locating the source of anomalies, a clear basis can be provided for rapid diagnosis and targeted optimization of abnormal problems. This can effectively reduce the frequency of false alarms on cross-border e-commerce platforms and greatly reduce maintenance costs and operating efficiency of cross-border e-commerce platforms.

[0116] Reference Manual Attached Figure 2 , showing a structural schematic diagram of a cross-border e-commerce big data anomaly detection system provided by the present invention.

[0117] The present invention also provides a cross-border e-commerce big data anomaly detection system 20, which is applied to the above-mentioned cross-border e-commerce big data anomaly detection method, comprising:

[0118] Processor 201.

[0119] The memory 202 stores computer-readable instructions. When the computer-readable instructions are executed by the processor 201, the cross-border e-commerce big data anomaly detection method as described in the method embodiment is implemented.

[0120] The cross-border e-commerce big data anomaly detection system 20 provided by the present invention can execute the above-mentioned cross-border e-commerce big data anomaly detection method and achieve the same or similar technical effects. To avoid repetition, the present invention will not go into details.

[0121] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0122] In the present invention, by performing dictionary learning on the historical cross-border e-commerce transaction data acquired within a preset period, the feature dictionary describing the normal transaction behavior of the historical cross-border e-commerce transaction data and the original reconstruction error of the historical cross-border e-commerce transaction data are determined in combination with the simulated annealing algorithm, and then the feature dictionary learned by the dictionary is used to reconstruct the current cross-border e-commerce transaction data, and the abnormal data is determined by taking the original reconstruction error as the judgment basis of the abnormal data, effectively capturing the normal mode of the historical transaction data, avoiding falling into the local optimum, and generating a more globally representative feature dictionary. Combined with the feature dictionary, the original reconstruction error is used as the benchmark to detect abnormal data, ensuring that the abnormal detection is not disturbed by local data, with high accuracy and robustness, and can adapt to complex transaction scene changes, improve the accuracy of detection and the adaptability to dynamic anomalies. After determining the abnormal data, the contribution value of each transaction feature to the current reconstruction error is calculated by the Shapley value, and the transaction feature with the largest collinearity of the current reconstruction error of the abnormal data is determined as the output of the abnormal cross-border e-commerce transaction data, accurately quantifying the impact of each feature on the anomaly, and locating the key abnormal features. By accurately identifying the features that have the greatest impact on anomalies and precisely locating the source of anomalies, a clear basis can be provided for rapid diagnosis and targeted optimization of abnormal problems. This can effectively reduce the frequency of false alarms on cross-border e-commerce platforms and greatly reduce maintenance costs and operating efficiency of cross-border e-commerce platforms.

[0123] It should be understood that the processor in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0124] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0125] The above embodiments can be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0126] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0127] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0128] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0129] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0131] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0132] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0133] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0134] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program codes.

[0135] An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the cross-border e-commerce big data anomaly detection method as described in the method embodiment is implemented.

[0136] A computer-readable storage medium provided by the present invention can implement the steps and effects of the cross-border e-commerce big data anomaly detection method of the above method embodiment. To avoid repetition, the present invention will not go into details.

[0137] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0138] In the present invention, by performing dictionary learning on the historical cross-border e-commerce transaction data acquired within a preset period, the feature dictionary describing the normal transaction behavior of the historical cross-border e-commerce transaction data and the original reconstruction error of the historical cross-border e-commerce transaction data are determined in combination with the simulated annealing algorithm, and then the feature dictionary learned by the dictionary is used to reconstruct the current cross-border e-commerce transaction data, and the abnormal data is determined by taking the original reconstruction error as the judgment basis of the abnormal data, effectively capturing the normal mode of the historical transaction data, avoiding falling into the local optimum, and generating a more globally representative feature dictionary. Combined with the feature dictionary, the original reconstruction error is used as the benchmark to detect abnormal data, ensuring that the abnormal detection is not disturbed by local data, with high accuracy and robustness, and can adapt to complex transaction scene changes, improve the accuracy of detection and the adaptability to dynamic anomalies. After determining the abnormal data, the contribution value of each transaction feature to the current reconstruction error is calculated by the Shapley value, and the transaction feature with the largest collinearity of the current reconstruction error of the abnormal data is determined as the output of the abnormal cross-border e-commerce transaction data, accurately quantifying the impact of each feature on the anomaly, and locating the key abnormal features. By accurately identifying the features that have the greatest impact on anomalies and precisely locating the source of anomalies, a clear basis can be provided for rapid diagnosis and targeted optimization of abnormal problems. This can effectively reduce the frequency of false alarms on cross-border e-commerce platforms and greatly reduce maintenance costs and operating efficiency of cross-border e-commerce platforms.

[0139] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0140] There are a few points to note:

[0141] (1) The drawings of the embodiments of the present invention only relate to the structures related to the embodiments of the present invention, and other structures may refer to the general design.

[0142] (2) For the sake of clarity, in the drawings used to describe the embodiments of the present invention, the thickness of the layers or regions is exaggerated or reduced, that is, these drawings are not drawn according to the actual scale. It is understood that when an element such as a layer, film, region or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element or there may be intermediate elements.

[0143] (3) In the absence of conflict, the embodiments of the present invention and the features therein may be combined with each other to obtain new embodiments.

[0144] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A cross-border e-commerce big data anomaly detection method, characterized in that: include: S1: Obtain historical cross-border e-commerce transaction data including different transaction characteristics within a preset period; S2: reconstructing the historical cross-border e-commerce transaction data based on dictionary learning with the goal of minimizing the reconstruction error of the historical cross-border e-commerce transaction data, and determining a feature dictionary describing normal transaction behaviors of the historical cross-border e-commerce transaction data and an original reconstruction error of the historical cross-border e-commerce transaction data in combination with a simulated annealing algorithm; S3: Get current cross-border e-commerce transaction data; S4: reconstructing the current cross-border e-commerce transaction data using the feature dictionary to determine a current reconstruction error of the current cross-border e-commerce transaction data; S5: If the current reconstruction error is greater than the original reconstruction error, it is determined that the current cross-border e-commerce transaction data is abnormal, and the process proceeds to step S6; otherwise, the process returns to step S3; S6: Calculate the contribution of each transaction feature to the current reconstruction error through the Shapley value; S7: Output the transaction feature corresponding to the maximum contribution value as abnormal cross-border e-commerce transaction data.

2. The cross-border e-commerce big data anomaly detection method according to claim 1 is characterized in that: The transaction characteristics include order amount, payment time interval, number of order items, return volume, cross-border e-commerce platform visits, cross-border e-commerce platform bounce rate and cross-border e-commerce platform order transaction volume.

3. The cross-border e-commerce big data anomaly detection method according to claim 1 is characterized in that: The method for determining the preset period specifically includes: S101: Obtain historical cross-border e-commerce transaction data of the cross-border e-commerce platform; S102: Standardizing the historical cross-border e-commerce transaction data: Among them, x(t) represents the historical cross-border e-commerce transaction data at time t, x′(t) represents the standardized x(t), μ and σ represent the mean and variance of x(t), respectively; S103: Perform Fourier transform on the historical cross-border e-commerce transaction data to obtain frequency domain features of the historical cross-border e-commerce transaction data: X′(f)=FFT(x′(t)) Wherein, FFT stands for Fast Fourier Transform, X′(f) stands for the energy distribution of x′(t) at different frequencies f, i.e., the frequency domain signal obtained after x′(t) is transformed by FFT; S104: Determine the preset period in the frequency domain features in combination with the spectrum entropy value of the frequency domain signal: f main =argmax f (|X'(f)|·(1-H f )) Among them, T main Indicates the preset period, f main Re(X'(f)) and Im(X'(f)) represent the real part and imaginary part of the frequency domain signal, respectively. f |X'(f)| represents the value f that maximizes the frequency domain signal amplitude |X'(f)|, log represents the logarithmic function, P(f) represents the probability distribution of f, H f Represents the spectral entropy value about f.

4. The cross-border e-commerce big data anomaly detection method according to claim 1 is characterized in that: The S2 specifically includes: S201: Initializing the feature dictionary and the sparse coefficient matrix with the goal of describing the historical cross-border e-commerce transaction data, wherein the sparse coefficient matrix is ​​used to describe the sparse representation of the historical cross-border e-commerce transaction data in the feature dictionary: X≈Dα X=[x1,x2,…,x n ] D=[d1,d2,…,d k ] α＝[α1,α2,…,α n ] Among them, X represents the matrix composed of historical cross-border e-commerce transaction data, D represents the feature dictionary, α represents the sparse coefficient matrix, and x i represents the i-th sample data in the historical cross-border e-commerce transaction data, i = 1, 2, ..., n, n represents the total number of sample data, d j The feature vector representing the jth transaction feature in the historical cross-border e-commerce transaction data, j = 1, 2, ..., k, k represents the total number of transaction features, α i Represents x i Sparse representation on D; S202: Taking minimizing the reconstruction error of the historical cross-border e-commerce transaction data as the update termination target, and using the update termination target as the energy function of the simulated annealing algorithm to update the feature dictionary and the sparse coefficient matrix, the update termination target is specifically: Among them, min means taking the minimum value, represents the square of the Frobenius norm, ||||1 represents the L1 norm, and λ represents the sparsity weight coefficient; S203: Calculate the original reconstruction error according to the updated feature dictionary and the sparse coefficient matrix: Among them, e baseline represents the original reconstruction error, and ||||2 represents the L2 norm.

5. The cross-border e-commerce big data anomaly detection method according to claim 4 is characterized in that: The S202 specifically includes: S2021: Initialize annealing temperature T and energy function value: Where E represents the energy function value with feature dictionary D and sparse coefficient matrix α; S2022: Add random perturbations to feature dictionaries and sparse coefficient matrices: D′=D+ΔD α′=α+Δα Among them, ΔD and Δα represent random feature dictionary perturbations and random sparse coefficient matrix perturbations, respectively, and D′ and α′ represent feature dictionary and sparse coefficient matrix after adding random perturbations, respectively; S2023: Calculate the energy function value after adding random perturbations: Among them, E′ represents the energy function value after adding random perturbations; S2024: The energy function difference between the two energy function values ​​calculated: ΔE=E′-E Among them, ΔE represents the energy function difference; S2025: If the absolute value of the energy function difference is less than the preset absolute value, proceed to step S2028; otherwise, proceed to step S2026; S2026: Update the feature dictionary and the sparse coefficient matrix according to the energy function difference: r t+1 =μr t (1-r t ),μ∈[3.5,4] Where exp represents the natural exponential function, μ represents the control parameter that controls the chaotic characteristics of random numbers, and r t and r t+1 Respectively represent the random numbers of the t-th update process and the t+1-th update process; S2027: Lower the annealing temperature and return to step S2022: T t+1 =T t ·γ t Among them, T t and T t+1 They represent the annealing temperatures of the tth update and the t+1th update process, γ0 represents the initial temperature attenuation coefficient, and γ t represents the temperature attenuation coefficient at the tth update, and log represents the logarithmic function; S2028: Output the current feature dictionary and the current sparse coefficient matrix as optimization results.

6. The cross-border e-commerce big data anomaly detection method according to claim 1 is characterized in that: The current reconstruction error is calculated as follows: Among them, X current represents the matrix composed of current cross-border e-commerce transaction data, α current Reconstruct X using D current The sparse coefficient matrix, e current Represents the current reconstruction error.

7. The cross-border e-commerce big data anomaly detection method according to claim 1 is characterized in that: The contribution value is calculated as follows: Among them, φ i represents the Shapley value of the i-th transaction feature to the current reconstruction error, i.e., the contribution value. N represents the set of all transaction features. {i} represents any transaction feature set that does not include transaction feature i, |S| represents the number of transaction features in transaction feature set S, |N| represents the number of transaction features in transaction feature set N, ! represents cumulative multiplication, E(S) represents the current reconstruction error after sparse reconstruction using S, and E(S∪{i}) represents the current reconstruction error recalculated after adding feature i to feature subset S.

8. The cross-border e-commerce big data anomaly detection method according to claim 1 is characterized in that: After S7, the method further includes: Issue early warning information about the abnormal cross-border e-commerce transaction data.

9. A cross-border e-commerce big data anomaly detection system, characterized in that: include: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the cross-border e-commerce big data anomaly detection method as described in any one of claims 1 to 8 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the cross-border e-commerce big data anomaly detection method as described in any one of claims 1 to 8 is implemented.

Images