Non-corresponding auditing system and method based on big data

Abstract

The invention relates to the technical field of inventory order auditing management, in particular to a big data-based non-correspondence auditing system and method.The big data-based non-correspondence auditing system comprises a data preprocessing unit, a data preliminary matching unit and a non-correspondence auditing unit. The data preliminary matching unit receives and processes data by using a message queue, partitions, de-duplicates and sorts the data, puts the data into a to-be-queued queue after preliminary matching by using a hash table structure, the non-corresponding auditing unit extracts features from historical data, establishes a dynamic threshold model through kernel density estimation, weights and fuses a threshold according to a data traffic load, and finally obtains a data stream; the deep learning model processes multi-dimensional features to output a normal delay probability to discriminate fuzzy data, and finally, a postponing area is set to review the fuzzy data according to a dynamic period to determine a final type, so that efficient data processing and accurate auditing are realized, data consistency is greatly improved, and enterprise operation decision scientificity is powerfully guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of inventory order review management, and in particular to a non-correspondence review system and method based on big data. Background Art

[0002] Inventory order review management is an important technology. In large e-commerce companies, the daily order volume is tens of thousands, the dynamic changes in commodity inventory are complex, the order system receives orders from multiple platforms, and inventory management covers warehouses in many locations. During the peak season, the order volume increases dramatically and inventory allocation is frequent, which makes it easy for orders and inventory data to not correspond.

[0003] In today's digital business wave, enterprise operations are highly dependent on the coordinated accuracy of order and inventory data. However, the disadvantages of the traditional data audit model are prominent. Manual audit is unable to cope with massive data. It is not only inefficient, but also prone to frequent errors due to fatigue and negligence, which poses hidden dangers to enterprise operations. The early automated audit system based on static thresholds, although efficient, lacks flexibility and is difficult to adapt to business peak and valley fluctuations and market dynamic changes. In complex business scenarios, fixed thresholds cannot accurately distinguish normal delays from abnormal data, often leading to misjudgments and missed judgments, resulting in overselling, out-of-stock, and inventory redundancy. In order to solve this problem, we provide a non-corresponding audit system and method based on big data. Summary of the invention

[0004] The purpose of the present invention is to provide a non-corresponding review system and method based on big data to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above purpose, a non-corresponding review system based on big data is provided, including a data pre-processing unit, a data preliminary matching unit and a non-corresponding review unit;

[0006] The data preprocessing unit adds timestamps and global identifiers to the data in the order center and the inventory management center, and classifies the order data and inventory data, extracts features, and then uses a hash algorithm to encode the features;

[0007] The data preliminary matching unit sets up a message queue to receive data change information sent by the order center and the inventory management center, and performs partition processing. After the consumer end removes duplicates based on the global identifier and sorts by timestamp, it uses a hash table data structure within a preset sliding window time, with the product number as the key and the difference between the sales and outbound quantities as the value to perform preliminary matching, and marks the data according to the matching result and puts it into the data queue to be checked;

[0008] The non-corresponding audit unit collects the normal delay distribution of sales and inventory data in historical data, uses kernel density estimation to build a dynamic threshold model, and according to the current data flow, adopts the weighted average method to fuse the historical data statistical threshold and the real-time impact threshold to obtain the final dynamic threshold when the system load changes, and builds a deep learning analysis model. For the data in the data queue to be checked, the probability that the output data is a normal delay is calculated, and the fuzzy data is judged based on the probability. Finally, the fuzzy data is placed in a preset delayed data suspension area, and an audit cycle is set. The difference between the sales and inventory quantities of the fuzzy data in the suspension area is recalculated and compared with the dynamic threshold, and the final type of the data is determined based on the comparison result.

[0009] As a further improvement of the technical solution, the data preliminary matching unit includes a partition processing module, and the method for performing partition processing in the partition processing module is as follows:

[0010] A hash function is constructed based on the data source, change type and commodity category information. The data source is the order center and the inventory management center. The data change type includes addition, modification and deletion. The commodity category is coded according to the classification system. The data source and change type are assigned and converted into binary strings.

[0011] The number of partitions is set according to the estimated amount of data, and the data volume of each partition is monitored in real time. It is compared with the standard deviation of the partition data volume, and then the new number of partitions is calculated based on the comparison result. Finally, the open addressing method is used for secondary detection processing.

[0012] As a further improvement of the present technical solution, the data preliminary matching unit includes a structure optimization module, and the structure optimization module is used to optimize the hash table data structure, as follows:

[0013] The open addressing method is combined with secondary detection to handle hash collisions. The hash collision refers to the situation in the hash table when two or more input data obtain the same hash value after being processed by the hash function and try to be stored in the same location;

[0014] When a conflict occurs, the new storage location is determined based on the secondary detection formula, and the hash table is rehashed regularly, and the hash function parameters and table size are dynamically adjusted based on the current data distribution.

[0015] As a further improvement of the technical solution, the structure optimization module uses a hash table data structure to perform preliminary matching with the product number as the key and the difference between the sales and outbound quantities as the value, as follows:

[0016] When using the hash table for preliminary matching, a Bloom filter pre-screening is introduced to map the historical sales and outbound data to the Bloom filter based on the product number hash. The Bloom filter stores data existence information in a bit array.

[0017] When the consumer side processes data, it first queries through the Bloom filter. If it does not exist, it is directly excluded. If it does exist, it is matched in the hash table. At the same time, for the hash table matching process, multi-threaded parallel processing technology is used to divide thread tasks according to product categories and concurrently calculate difference matching;

[0018] The data within the sliding window time is segmented according to the timestamp, and each segment is processed by an independent thread. The results of each segment are summarized, integrated and sorted, and then marked into the data queue to be checked according to the matching results.

[0019] As a further improvement of the technical solution, the non-corresponding review unit includes a model building module, and the steps of building a dynamic threshold model by the model building module are as follows:

[0020] Extract multi-dimensional features from historical sales and inventory data to construct a sample data set, use Gaussian kernel function to perform kernel density estimation, and determine the optimal bandwidth value through cross-validation. The bandwidth is a parameter used to control the smoothness of the estimation curve in kernel density estimation.

[0021] The threshold intervals at different confidence levels are determined based on the estimated density distribution curve. As new data is input, the sample data set and kernel density estimation results are dynamically updated, and the threshold intervals are adjusted in real time.

[0022] As a further improvement of the technical solution, the non-corresponding review unit includes a weighted average module, and the weighted average fusion threshold method in the weighted average module is as follows:

[0023] Setting historical data statistical thresholds, real-time impact thresholds and system load factors. The real-time impact thresholds are calculated and determined based on current data traffic and system load fluctuations, and weights are determined based on the system load factors. The final dynamic thresholds are then calculated based on the determined weights.

[0024] The system load factor is quantified by CPU usage, memory occupancy, and task queue length indicators.

[0025] As a further improvement of the technical solution, the non-corresponding audit unit includes an audit analysis module, and the feature enhancement method of the deep learning analysis model in the audit analysis module is as follows:

[0026] Performing deep feature engineering processing on the multi-dimensional features of the input data to be checked, wherein the multi-dimensional features include commodity sales fluctuation cycle features and commodity association graph features;

[0027] The commodity sales fluctuation cycle feature mines the hidden cycle information of the time series through discrete Fourier transform, sets the number of transformation points and sampling frequency, and the sampling frequency is adjusted according to the data time interval. The commodity association graph feature learns the impact of the association relationship between commodities on delay based on the co-occurrence matrix and the graph convolution network. The co-occurrence matrix is ​​constructed based on the number of orders in which commodities appear together in the past 30 days. The graph convolution network has 3 layers, and the multi-feature fusion is input into the deep learning model after pre-training and dimension reduction by the multi-layer perceptron.

[0028] The deep learning model outputs the probability that the data is a normal delay, and judges the fuzzy data based on the probability.

[0029] As a further improvement of the technical solution, the audit analysis module recalculates the difference between sales and inventory quantity for the fuzzy data in the suspension area and compares it with the dynamic threshold value, specifically as follows:

[0030] Construct a delayed data temporary storage area based on a hybrid storage architecture of memory and disk. The memory uses a high-speed cache area to temporarily store fuzzy data, and the disk is used to store historical fuzzy data. The memory cache area capacity and the fuzzy data unique identifier are set, and the original data features, the first analysis probability, and the entry time stamp are associated to build an index.

[0031] Build comprehensive evaluation indicators for data flow, system load and fuzzy data volume, and dynamically set the review cycle based on the comprehensive evaluation indicators. At the review cycle node, extract fuzzy data from the suspension area based on identification and index;

[0032] After extracting the fuzzy data, recalculate the difference between sales and inventory quantities, compare the difference with the above dynamic threshold, and output abnormal data based on the comparison.

[0033] The second object of the present invention is to provide a method for operating the above-mentioned non-corresponding review system based on big data, comprising the following steps:

[0034] S1. Extract data from the order center and inventory management center, add timestamps and global identifiers to them, classify order and inventory data according to business rules, extract commodity categories, quantities, and time features, and use hash algorithms to encode features;

[0035] S2. Build a message queue to receive data change information. Partition processing builds a hash function to allocate partitions based on data source, change type, and product category. Monitor and adjust the number of partitions and handle conflicts. The consumer end removes duplicates based on global identifiers and sorts by timestamps. A hash table structure is used for preliminary matching in the sliding window, and the marked results are included in the queue of data to be checked.

[0036] S3, extract multi-dimensional features from historical data to construct a sample set, and use Gaussian kernel function kernel density estimation, cross-validation to determine the bandwidth, obtain the threshold range, and then dynamically update the sample and threshold with new data, calculate the statistical threshold from historical data, and obtain the real-time threshold based on real-time data flow and system load fluctuations, and finally weighted fusion to obtain the final threshold;

[0037] S4. After deep processing of the multi-dimensional features of the input data to be checked, the model is input and the normal delay probability is output. The fuzzy data is judged based on the probability threshold, and a memory-disk hybrid temporary storage area is built to store the fuzzy data. The memory cache capacity and identification index management are set. The audit cycle is set according to the data flow, system load and fuzzy data volume indicators. The data is extracted at the audit cycle node, the sales and inventory difference is recalculated and compared with the final threshold, and finally processed according to the results.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The data preliminary matching unit receives data with the help of the message queue, and performs preliminary matching through partitioning, deduplication, sorting and hash table structure to quickly screen and organize the data, laying a solid foundation for subsequent review and greatly improving data processing efficiency. On this basis, the non-corresponding review unit uses kernel density estimation and weighted fusion to build a dynamic threshold model. The deep learning model processes multi-dimensional features to discriminate fuzzy data, accurately distinguishes normal delays from abnormal situations, and reduces the risk of misjudgment. The combination of the two forms a precise review link to ensure the consistency of order and inventory data. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is an overall block diagram of the present invention;

[0041] Figure 2 It is the overall flow chart of the present invention.

[0042] The meaning of each number in the figure is:

[0043] 1. Data preprocessing unit; 2. Data preliminary matching unit; 21. Partition processing module; 22. Structure optimization module; 3. Non-corresponding audit unit; 31. Model building module; 32. Weighted average module; 33. Audit analysis module. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] Example 1

[0046] The present invention provides a non-corresponding audit system based on big data, please refer to Figure 1 As shown, it includes a data preprocessing unit 1, a data preliminary matching unit 2 and a non-corresponding review unit 3;

[0047] The data preprocessing unit 1 adds timestamps and global identifiers to the data in the order center and the inventory management center, and classifies the order data and inventory data, extracts features, and then uses a hash algorithm to encode the features;

[0048] The data preliminary matching unit 2 builds a message queue to receive data change information sent by the order center and the inventory management center, and performs partition processing. After the consumer end removes duplicates based on the global identifier and sorts by timestamp, it uses the hash table data structure within the preset sliding window time, with the product number as the key and the difference between the sales and outbound quantities as the value to perform preliminary matching, and marks the data according to the matching results and puts it into the data queue to be checked;

[0049] The data preliminary matching unit 2 includes a partition processing module 21. The method for performing partition processing in the partition processing module 21 is as follows:

[0050] A hash function is constructed based on the data source, change type, and product category information. For the data source, the order center is set to 0, the inventory management center is set to 1, the data change type is 00 for addition, 01 for modification, and 10 for deletion. The product category is coded according to the classification system, such as 001 for food and 010 for electronic products. This combined information is converted into a binary string. For example, the code for newly added food data in the order center is "0001001". The weighted sum method is used to calculate the hash value. Set the data source weight , Change Type And product categories , calculate the hash function for the above encoding, where the expression of the hash function is ;in is the number of partitions, initially set to 10, and the hash value is 5, corresponding to partition 5. This function considers multiple key factors to ensure that data of different properties are evenly hashed.

[0051] Initially determine the number of partitions based on the estimated amount of data , real-time monitoring of the data volume of each partition, if the standard deviation of the partition data volume exceeds the mean , triggering partition adjustment, such as the amount of data in partition 3 exceeds ,calculate ;in, is the maximum partition data size, is the average data volume, is the adjustment factor, Recalculate the hash value and partition mapping for the new number of partitions, achieve load balancing through data migration, ensure that the system continues to process data efficiently and adapt to dynamic business changes. Conflicts are inevitable in hash calculations, and open addressing method is used for secondary detection and processing.

[0052] The data preliminary matching unit 2 includes a structure optimization module 22, which is used to optimize the hash table data structure as follows:

[0053] When new data is inserted into the hash table and a conflict occurs, according to the secondary detection formula ; Determine the storage location and set the initial hash value , the first conflict , then the new position ,like Still conflicting, Further exploration, ; Continue until an idle position is found, compared to linear detection , which disperses conflicting data. Linear detection conflicts are prone to data aggregation. Secondary detection makes data distribution more uniform due to the square step length. It reduces continuous empty slot detection during search and reduces the average search length. , significantly improving efficiency, showing its advantages in processing large amounts of data and accelerating data reading and writing responses.

[0054] Rehashing is started every time 1,000 data are processed. The distribution frequency of key features of the current data is first counted, such as the first digit distribution of the product number and the time interval distribution of data changes. Weights are assigned according to the importance of the features, and feature distribution vectors are constructed. The hash function parameters are adjusted accordingly, such as fine-tuning the data source weight or resetting the change type code. At the same time, the size of the new table is determined based on the amount of data and the uniformity of distribution. , if the proportion of data volume of a certain product category increases ,calculate ; is the data volume variance, is the mean, , and then re-hash the original table data according to the new function and table to ensure that the hash table structure adapts to the dynamic changes of data, maintains the efficiency of search and insertion, ensures the stable operation of the system, and avoids the degradation of hash performance due to the evolution of data characteristics.

[0055] The structure optimization module 22 uses a hash table data structure to perform a preliminary match with the product number as the key and the difference between the sales and outbound quantities as the value, as follows:

[0056] Extract massive product ID data from the historical sales and delivery database, calculate the hash value based on multiple selected hash functions, and set the length of the Bloom filter bit array to be , the number of hash functions is , according to the quantity and expected misjudgment rate Determine the parameters, the formula is ; ; Set the bit array position corresponding to each hash value to 1 to complete the construction of the Bloom filter, which compactly stores data existence information. When the consumer receives new data, the product number is checked by the Bloom filter by calculating the hash value through the same hash function. If all corresponding bits are not all 1, the data must not exist and is directly excluded, which greatly reduces the number of queries in the hash table. If all are 1, it exists and is then accurately matched in the hash table. Based on the efficient characteristics of the Bloom filter, a large amount of irrelevant data can be quickly screened out in the big data scenario, significantly improving the efficiency of the preliminary matching front-end link and reducing system resource consumption and processing delay.

[0057] Divide thread tasks according to product categories, build a thread pool to allocate processing threads, such as Threads divide task queues according to product categories, and tasks of different categories are included in corresponding queues. When threads are idle, they take tasks from queues. Task data is obtained within threads, and the difference between sales and outbound quantities is matched based on hash tables for product numbers, making full use of multi-core CPU resources, greatly improving matching calculation speed, and reducing overall matching time.

[0058] The data in the sliding window time is segmented according to the timestamp, and the length of each segment is Dynamically adjust according to data flow and processing capacity, such as high flow Short, low flow Length, set to ;in is the sliding window length, The number of segments is calculated, and an independent processing thread is created for each segment. The data in the segment is read in the thread, the difference between the sales and outbound quantities of each product number is calculated and matched with the hash table, and the matching results and corresponding product numbers are recorded. After the processing of each segment is completed, the main thread summarizes and integrates the results, sorts them according to timestamps, and marks the data according to the matching results and includes them in the queue of data to be checked. This segmented processing adapts to the time characteristics of the data and the dynamic load of the system, avoids memory overflow and processing delays caused by centralized processing of large amounts of data, ensures orderly review of data in different time periods, improves the system's ability to process complex data sequences and stability, ensures the continuity and accuracy of the data review process, and optimizes the performance of the entire system data processing process and resource utilization efficiency.

[0059] The non-corresponding audit unit 3 collects the normal delay distribution of sales and inventory data in historical data, uses kernel density estimation to build a dynamic threshold model, and according to the current data flow, adopts the weighted average method to fuse the historical data statistical threshold and the real-time impact threshold to obtain the final dynamic threshold when the system load changes, and builds a deep learning analysis model. For the data in the data queue to be checked, the output data is the probability of normal delay, and the fuzzy data is judged according to the probability. Finally, the fuzzy data is placed in a preset delayed data suspension area, and the audit cycle is set. The difference between the sales and inventory quantities of the fuzzy data in the suspension area is recalculated and compared with the dynamic threshold, and the final data type is determined based on the comparison results.

[0060] The non-corresponding review unit 3 includes a model building module 31. The steps of building a dynamic threshold model by the model building module 31 are as follows:

[0061] We construct a sample set by mining multi-dimensional features from massive historical sales and inventory data. We calculate the time interval from order generation to inventory change. Product popularity is quantified based on weights of sales, search volume, and evaluation volume. Seasonal factors are coded according to the industry's seasonal fluctuations, such as 1 for summer and -1 for winter. We group data by time slices, extract feature vectors from each group, and integrate them to form a sample data set to ensure that business cycle changes are covered, laying the foundation for the threshold model. This multi-dimensional feature extraction comprehensively captures data characteristics, allowing the sample set to accurately reflect business scenarios and improve model applicability. Compared with single-dimensional feature models, it has a strong ability to analyze complex data patterns, laying a solid foundation for subsequent accurate threshold estimation and optimizing the accuracy of data mismatch audits.

[0062] Gaussian kernel function The kernel density of each dimension of the sample data set is estimated separately, and the cross-validation method is used to find the optimal bandwidth , divide the sample set into training and validation subsets to traverse Calculate the likelihood value of the validation set in the value range and take the maximum likelihood , kernel density estimation of sales data, different The difference in the lower density curve is large and accurate It makes the curve fit better, highlights the data distribution characteristics, provides a reliable basis for threshold determination, improves the accuracy of density estimation, enhances the robustness of the threshold model, ensures that it can accurately depict the core trend of data distribution, effectively distinguish normal and abnormal delays, and reduce misjudgment.

[0063] According to the kernel density estimation curve, the confidence interval is determined as the threshold interval. As new data flows in, the sample set is updated according to the time window, old data is removed and new data is included. The new curve and threshold are obtained by re-kernel density estimation. The time window is daily. The delayed mean of inventory change is set to , Standard Deviation , Confidence interval When a new order pattern emerges or the season changes, the data distribution migration is captured dynamically to maintain the effectiveness of the threshold, ensure that the corresponding audit system accurately adapts to business changes, responds to data anomalies in a timely manner, improves the system's real-time monitoring and risk warning capabilities, and consolidates data quality management efficiency.

[0064] The non-corresponding review unit 3 includes a weighted average module 32. The weighted average fusion threshold method in the weighted average module 32 is as follows:

[0065] Mining normal delay data samples from the historical sales and inventory data accumulated over a long period of time in the system, calculating the mean and standard deviation of the difference between sales and outbound quantities for each subset based on the time period and product category, and determining the threshold based on the business risk tolerance , business risk tolerance reflects the maximum risk level that an enterprise can bear when facing potential losses or uncertainties. For example, according to the normal distribution characteristics of electronic products, Confidence interval Define the normal delay range, and the boundary value is , determine the historical threshold as the foundation for integration, reflect past business rules, help the system identify common normal delay patterns, stabilize the initial audit benchmark, reduce the risk of abnormal misjudgment, ensure audit accuracy and efficiency during the data stability period, and anchor a reliable reference for subsequent dynamic adjustments.

[0066] Build a data flow monitoring module to capture the order and inventory data flow rate, flow peak and fluctuation coefficient in real time. The system load monitoring module collects CPU usage, memory usage and task queue length indicators to quantify load fluctuations. Triggering real-time impact threshold Calculation is based on the traffic load association rule base and real-time parameters. The traffic load association rule base is obtained by mapping predefined traffic load and delay threshold. For example, when the traffic is high and the load is high, the power function is used to calculate the delay threshold. Calculate ,in Expressed as flow rate, Expressed as load, It is an empirical parameter that reflects the impact of environmental changes on latency in real time, ensures that the threshold can flexibly adapt to the working conditions, enhances the real-time responsiveness of the audit system, accurately captures transient anomalies, and ensures audit sensitivity and accuracy during business peaks.

[0067] System load factor Calculated based on CPU usage , memory usage And the proportion of task queue length Weighted, set ; is the threshold value, , normalized to 0-1, when the load is low Near 0, when the load is high Nearly 1, based on Weight , , calculate the final dynamic threshold The fusion is dynamically weighted according to the load, balancing historical experience and real-time conditions, accurately weighing the thresholds under different loads, avoiding the limitations of a single threshold, improving the environmental adaptability of the audit system, ensuring that the audit of data under various working conditions is accurate and efficient, and optimizing the scientific nature of data processing decisions.

[0068] The non-corresponding audit unit 3 includes an audit analysis module 33. The feature enhancement method of the deep learning analysis model in the audit analysis module 33 is as follows:

[0069] For the time series of commodity sales data in the data queue to be checked, discrete Fourier transform is used to mine the fluctuation period characteristics. First, the sales data series Discrete Fourier transform calculation is performed, and the calculation formula is: ; , convert it from the time domain to the frequency domain and get the spectrum Then, the significant periodic component is determined according to the spectrum amplitude, and the amplitude threshold is set to , select an amplitude exceeding The frequency components corresponding to the period ; as a feature, among which To index the significant frequency components, the Hanning window function is used to enhance stability and noise resistance. After weighted processing of the original data, discrete Fourier transform calculation is performed. This feature extraction accurately captures the cyclical law of sales fluctuations, provides key information in the time dimension for the model, helps it identify delay patterns caused by cyclical factors, improves the accuracy of fuzzy data discrimination, and optimizes the adaptability of data audit to market rhythm.

[0070] Constructing product co-occurrence matrix based on historical order data , statistics of goods and Occurrences in the same order , let the order set ,but ;in is the Dirac function, normalizing the co-occurrence matrix ; The adjacency matrix represents the strength of commodity association, and the graph convolutional network is constructed. Layer, Layer node feature matrix Update based on formula ; is an adjacency matrix with self-loops, is the degree matrix, is the activation function, The product association graph features are represented by the layer weight matrix, and the final features of the nodes are used to represent the features of the product association graph. Through multi-layer convolution learning, the influence of the complex association structure of products on the delayed transmission of sales and inventory is injected into the model with topological structure information, which strengthens the semantic mining of data association, improves the ability to classify fuzzy data under complex business relationships, accurately identifies abnormal delays caused by related factors, and improves the depth and breadth of the audit logic.

[0071] The product sales fluctuation cycle characteristics and product association graph characteristics are input into the deep learning model, where the deep learning model is a fusion architecture of convolutional neural network and long short-term memory network. The convolution layer extracts local features with convolution kernels of different sizes, the pooling layer compresses data to reduce dimension, and the long short-term memory network captures data time series based on the characteristics. The model training is based on a large amount of historical audit data and corresponding features, and the cross entropy loss function is optimized. The historical audit data includes clear normal delay and abnormal data annotations, and the cross entropy loss function is optimized to learn the mapping of features to normal delay probabilities. In the inference stage, the normal delay probability value between 0 and 1 is output for the input data to be checked. This probability reflects the degree of matching of the model to the data feature pattern based on the learned knowledge and the confidence in the normal delay distribution, laying the foundation for the subsequent accurate judgment of the nature of fuzzy data, improving the level of audit automation and intelligence, and accurately identifying potential anomalies with data-driven.

[0072] Set the dynamic probability threshold interval , dynamically adjusted according to the data traffic load, where the data traffic includes the number of orders received per second, the number of inventory change records per second, and the data transmission bandwidth utilization rate. If the data traffic does not exceed the average of the three during the continuous monitoring period, it is determined that the data traffic is stable and low-loaded. , If, during the continuous monitoring period, one of the data traffic items exceeds the average of the three, it is considered as a data traffic peak. , , when the probability of normal delay of the data to be checked is Fuzzy data is determined to be within the interval. Such data features are complex and difficult to judge based on a single threshold. The fuzzy data is marked and recorded with feature vectors, including product number, timestamp, probability value, feature values ​​of each dimension, etc., for further analysis of the data set. Based on the probability interval, suspected abnormal data is screened and focused to avoid misjudgment and missed judgment, balance the rigor and efficiency of the audit, accurately locate the data that needs in-depth audit, and provide accurate samples for subsequent temporary suspension area processing and root cause tracing.

[0073] The audit analysis module 33 recalculates the difference between the sales and inventory quantities for the fuzzy data in the suspension area and compares it with the dynamic threshold value, as follows:

[0074] Construct a delayed data suspension area based on a hybrid storage architecture of memory and disk. The memory uses a high-speed cache area to temporarily store recently active fuzzy data to improve access speed, and the disk stores large-capacity historical fuzzy data to ensure data integrity. Set the memory cache area capacity The disk storage area is dynamically expanded according to the growth of data volume, a unique identifier is established for fuzzy data, and an index table is established for easy retrieval by associating metadata such as original data characteristics, first analysis probability, and entry timestamp. According to the entry timestamp and data quality monitoring, when the cache capacity threshold is exceeded, the early data is transferred to the disk according to the first-in-first-out principle, and expired disk data is regularly cleaned up to optimize the use of storage resources. An efficient and persistent data management environment is provided for fuzzy data review, ensuring orderly storage and processing of data, accelerating the turnover of the review process, and improving the overall data operation efficiency and stability of the system.

[0075] Construct a comprehensive evaluation index of data flow, system load and fuzzy data volume. The specific formula is:

[0076] ;

[0077] in, is the relative peak ratio of real-time data traffic, , is the system load indicator, which is the weighted sum of CPU usage and memory usage. is the load threshold, is the amount of fuzzy data in the temporary area, is the average data volume, To adjust the coefficient, set the audit cycle according to the comprehensive evaluation index E ,like hour, Hour, hour, Hour, hour, Every hour, at the audit cycle node, fuzzy data is efficiently extracted from the suspension area based on identification and index, laying the foundation for re-evaluating the data status, ensuring that the audit rhythm adapts to the system operating conditions and data backlog, improving the timeliness of the audit and the rationality of resource utilization, balancing processing efficiency and accuracy, optimizing the management and control efficiency of the entire data audit process, and ensuring the continuous improvement of business data quality.

[0078] Recalculate the difference between sales and inventory quantities after extracting fuzzy data ,Will With dynamic threshold Compare, if Determine the data as normal delay, update the database mark and record the audit trail. If the data is determined to be abnormal, the traceability analysis process will be triggered to check the logs of each link of the order processing, inventory operation records, and system interface interaction data, locate the root cause of the abnormality and classify and mark it, process the data according to the final type, archive the normal data for analysis and decision-making, and generate repair instructions based on the root cause of the abnormal data to feedback the corresponding business module for rectification, ensure closed-loop management of data quality, accurately identify the nature of data, strengthen the system's fault diagnosis and repair capabilities, improve the reliability of business processes and the scientific nature of data-driven decision-making, and consolidate the system's core operational efficiency and data security defenses.

[0079] In the present invention, the data preprocessing unit 1 adds identification, classification, feature extraction and hash coding to the data, the data preliminary matching unit 2 uses the message queue to receive and process the data, and after partitioning, deduplication and sorting, the data is placed in the queue to be checked after preliminary matching with the hash table structure. The non-corresponding review unit 3 extracts features from historical data and builds a dynamic threshold model through kernel density estimation. The threshold is weighted according to the data traffic load, and the multi-dimensional features are processed by the deep learning model to output the normal delay probability to distinguish the fuzzy data. Finally, a suspension area is set to review the fuzzy data according to the dynamic period to determine the final type, so as to realize efficient data processing and accurate review, greatly improve data consistency, and effectively ensure the scientific nature of enterprise operation decision-making.

[0080] Example 2

[0081] The second object of the present invention is to provide a method for operating the above-mentioned non-corresponding review system based on big data, comprising the following steps:

[0082] S1. Extract data from the order center and inventory management center, add timestamps and global identifiers to them, classify order and inventory data according to business rules, extract commodity categories, quantities, and time features, and use hash algorithms to encode features;

[0083] S2. Build a message queue to receive data change information. Partition processing builds a hash function to allocate partitions based on data source, change type, and product category. Monitor and adjust the number of partitions and handle conflicts. The consumer end removes duplicates based on global identifiers and sorts by timestamps. A hash table structure is used for preliminary matching in the sliding window, and the marked results are included in the queue of data to be checked.

[0084] S3, extract multi-dimensional features from historical data to construct a sample set, and use Gaussian kernel function kernel density estimation, cross-validation to determine the bandwidth, obtain the threshold range, and then dynamically update the sample and threshold with new data, calculate the statistical threshold from historical data, and obtain the real-time threshold based on real-time data flow and system load fluctuations, and finally weighted fusion to obtain the final threshold;

[0085] S4. After deep processing of the multi-dimensional features of the input data to be checked, the model is input and the normal delay probability is output. The fuzzy data is judged based on the probability threshold, and a memory-disk hybrid temporary storage area is built to store the fuzzy data. The memory cache capacity and identification index management are set. The audit cycle is set according to the data flow, system load and fuzzy data volume indicators. The data is extracted at the audit cycle node, the sales and inventory difference is recalculated and compared with the final threshold, and finally processed according to the results.

[0086] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. The non-correspondence review system based on big data is characterized by: It includes a data preprocessing unit (1), a data preliminary matching unit (2), and a non-corresponding review unit (3); The data preprocessing unit (1) adds a timestamp and a global identifier to the data in the order center and the inventory management center, and classifies the order data and the inventory data, extracts features, and then uses a hash algorithm to encode the features; The data preliminary matching unit (2) builds a message queue for receiving data change information sent by the order center and the inventory management center, and performs partition processing. After the consumer end removes duplicates based on the global identifier and sorts by timestamp, it uses a hash table data structure within a preset sliding window time, uses the product number as the key, and the difference between the sales and outbound quantities as the value to perform preliminary matching, and marks the data according to the matching result and puts it into the queue of data to be checked; The non-corresponding audit unit (3) collects the normal delay distribution of sales and inventory data in historical data, uses kernel density estimation to build a dynamic threshold model, and uses a weighted average method to fuse the historical data statistical threshold and the real-time impact threshold according to the current data flow when the system load changes to obtain a final dynamic threshold, and builds a deep learning analysis model. For the data in the to-be-checked data queue, the probability that the output data is a normal delay is determined, and fuzzy data is judged based on the probability. Finally, the fuzzy data is placed in a preset delayed data suspension area, an audit cycle is set, the difference between the sales and inventory quantities of the fuzzy data in the suspension area is recalculated and compared with the dynamic threshold, and the final type of the data is determined based on the comparison result.

2. The big data-based non-correspondence review system according to claim 1, characterized in that: The data preliminary matching unit (2) comprises a partition processing module (21), and the method for performing partition processing in the partition processing module (21) is specifically as follows: A hash function is constructed based on the data source, change type and commodity category information. The data source is the order center and the inventory management center. The data change type includes addition, modification and deletion. The commodity category is coded according to the classification system. The data source and change type are assigned and converted into binary strings. The number of partitions is set according to the estimated amount of data, and the data volume of each partition is monitored in real time. It is compared with the standard deviation of the partition data volume, and then the new number of partitions is calculated based on the comparison result. Finally, the open addressing method is used for secondary detection processing.

3. The non-correspondence audit system based on big data according to claim 2 is characterized in that: The data preliminary matching unit (2) comprises a structure optimization module (22), and the structure optimization module (22) is used to optimize the hash table data structure, specifically as follows: The open addressing method is combined with secondary detection to handle hash collisions. The hash collision refers to the situation in the hash table when two or more input data obtain the same hash value after being processed by the hash function and try to be stored in the same location; When a conflict occurs, the new storage location is determined based on the secondary detection formula, and the hash table is rehashed regularly, and the hash function parameters and table size are dynamically adjusted based on the current data distribution.

4. The big data-based non-correspondence review system according to claim 3 is characterized by: The structure optimization module (22) uses a hash table data structure to perform a preliminary match using the product number as a key and the difference between the sales and delivery quantities as a value, specifically as follows: When using the hash table for preliminary matching, a Bloom filter pre-screening is introduced to map the historical sales and delivery data to the Bloom filter based on the product number hash. The Bloom filter stores data existence information in a bit array. When the consumer side processes data, it first queries through the Bloom filter. If it does not exist, it is directly excluded. If it does exist, it is matched in the hash table. At the same time, for the hash table matching process, multi-threaded parallel processing technology is used to divide thread tasks according to product categories and concurrently calculate difference matching; The data within the sliding window time is segmented according to the timestamp, and each segment is processed by an independent thread. The results of each segment are summarized, integrated and sorted, and then marked into the data queue to be checked according to the matching results.

5. The big data-based non-correspondence review system according to claim 1 is characterized by: The non-corresponding review unit (3) comprises a model building module (31), and the steps of building a dynamic threshold model in the model building module (31) are specifically as follows: Extract multi-dimensional features from historical sales and inventory data to construct a sample data set, use Gaussian kernel function to perform kernel density estimation, and determine the optimal bandwidth value through cross-validation. The bandwidth is a parameter used to control the smoothness of the estimation curve in kernel density estimation. The threshold intervals at different confidence levels are determined based on the estimated density distribution curve. As new data is input, the sample data set and kernel density estimation results are dynamically updated, and the threshold intervals are adjusted in real time.

6. The non-correspondence review system based on big data according to claim 5 is characterized by: The non-corresponding review unit (3) comprises a weighted average module (32), wherein the weighted average fusion threshold method in the weighted average module (32) is specifically as follows: Setting historical data statistical thresholds, real-time impact thresholds and system load factors. The real-time impact thresholds are calculated and determined based on current data traffic and system load fluctuations, and weights are determined based on the system load factors. The final dynamic thresholds are then calculated based on the determined weights. The system load factor is quantified by CPU usage, memory occupancy, and task queue length indicators.

7. The big data-based non-correspondence review system according to claim 1, characterized in that: The non-corresponding audit unit (3) comprises an audit analysis module (33), and the feature enhancement method of the deep learning analysis model in the audit analysis module (33) is specifically as follows: Performing deep feature engineering processing on the multi-dimensional features of the input data to be checked, wherein the multi-dimensional features include commodity sales fluctuation cycle features and commodity association graph features; The commodity sales fluctuation cycle feature mines the hidden cycle information of the time series through discrete Fourier transform, sets the number of transformation points and sampling frequency, and the sampling frequency is adjusted according to the data time interval. The commodity association graph feature learns the impact of the association relationship between commodities on delay based on the co-occurrence matrix and the graph convolution network. The co-occurrence matrix is ​​constructed based on the number of orders in which commodities appear together in the past 30 days. The graph convolution network has 3 layers, and the multi-feature fusion is input into the deep learning model after pre-training and dimension reduction by the multi-layer perceptron. The deep learning model outputs the probability that the data is a normal delay, and judges the fuzzy data based on the probability.

8. The big data-based non-correspondence review system according to claim 7, characterized in that: The audit analysis module (33) recalculates the difference between the sales and inventory quantities for the fuzzy data in the suspension area and compares it with the dynamic threshold value, specifically as follows: Construct a delayed data temporary storage area based on a hybrid storage architecture of memory and disk. The memory uses a high-speed cache area to temporarily store fuzzy data, and the disk is used to store historical fuzzy data. The memory cache area capacity and the fuzzy data unique identifier are set, and the original data features, the first analysis probability, and the entry time stamp are associated to build an index. Build comprehensive evaluation indicators for data flow, system load and fuzzy data volume, and dynamically set the review cycle based on the comprehensive evaluation indicators. At the review cycle node, extract fuzzy data from the suspension area based on identification and index; After extracting the fuzzy data, recalculate the difference between sales and inventory quantities, compare the difference with the above dynamic threshold, and output abnormal data based on the comparison.

9. A method for operating the non-correspondence review system based on big data according to any one of claims 1 to 8, characterized in that: The following steps are involved: S1. Extract data from the order center and inventory management center, add timestamps and global identifiers to them, classify order and inventory data according to business rules, extract commodity categories, quantities, and time features, and use hash algorithms to encode features; S2. Build a message queue to receive data change information. Partition processing builds a hash function to allocate partitions based on data source, change type, and product category. Monitor and adjust the number of partitions and handle conflicts. The consumer end removes duplicates based on global identifiers and sorts by timestamps. A hash table structure is used for preliminary matching in the sliding window, and the marked results are included in the queue of data to be checked. S3, extract multi-dimensional features from historical data to construct a sample set, and use Gaussian kernel function kernel density estimation, cross-validation to determine the bandwidth, obtain the threshold range, and then dynamically update the sample and threshold with new data, calculate the statistical threshold from historical data, and obtain the real-time threshold based on real-time data flow and system load fluctuations, and finally weighted fusion to obtain the final threshold; S4. After deep processing of the multi-dimensional features of the input data to be checked, the model is input and the normal delay probability is output. The fuzzy data is judged based on the probability threshold, and a memory-disk hybrid temporary storage area is built to store the fuzzy data. The memory cache capacity and identification index management are set. The audit cycle is set according to the data flow, system load and fuzzy data volume indicators. The data is extracted at the audit cycle node, the sales and inventory difference is recalculated and compared with the final threshold, and finally processed according to the results.

Images