High-concurrency transaction data processing system and method combined with edge computing

Abstract

The invention discloses a high-concurrency transaction data processing system and a high-concurrency transaction data processing method based on edge computing, so as to improve transaction data processing efficiency and security. According to the method, transaction data are simultaneously received from a plurality of user terminals through a multi-thread concurrent receiving queue of edge equipment, and strict integrity and legality verification is carried out on the data. Verified data is fragmented and intelligently distributed to a plurality of processing modules for parallel processing. And the edge device intelligently integrates processing results of the modules by adopting a real-time parallel reduction algorithm to generate transaction response data. And then, the response data is compressed by using a self-adaptive compression algorithm, the compressed response data is synchronized to the central server through a multi-encrypted secure transmission channel, and a synchronization state is fed back to the user terminal in real time. And after the data is successfully synchronized, the edge device safely deletes the local cache to ensure high efficiency, safety and reliability of data processing.

Description

Technical Field

[0001] The present invention relates to the field of edge computing data processing technology, and in particular to a high-concurrency transaction data processing system and method combined with edge computing. Background Art

[0002] With the rapid development of the digital economy, transaction data processing systems in the fields of Internet transactions, financial services, and e-commerce are facing unprecedented challenges. The performance bottleneck of the traditional centralized data processing architecture is becoming increasingly prominent when facing high-concurrency, low-latency transaction scenarios. Traditional centralized servers are difficult to handle a large number of concurrent requests at the same time, and are prone to problems such as system response delays and limited processing capabilities. When user requests surge, the system is prone to congestion and crashes, seriously affecting user experience and business continuity.

[0003] Traditional architecture also has many shortcomings in terms of data security and network transmission. The centralized architecture has the risk of single point failure. Once the central server is attacked or fails, the entire system will be paralyzed, and the data will face huge security risks. Large-scale data transmission requires a lot of bandwidth. Traditional transmission methods lack effective compression and security mechanisms, which increases the network burden and data transmission risks, and it is difficult to effectively improve resource utilization.

[0004] Faced with these challenges, edge computing technology has emerged, providing a new technical path to address the limitations of traditional data processing systems. By sinking computing, storage, and network resources to edge nodes close to data sources, the system's concurrent processing capabilities can be improved, latency can be reduced, data security can be improved, and network resource utilization can be optimized. However, existing edge computing solutions still have many limitations and have not yet fully resolved the complex technical challenges of high-concurrency transaction data processing. Summary of the invention

[0005] In view of this, an object of an embodiment of the present invention is to provide a high-concurrency transaction data processing system and method based on edge computing to solve the above-mentioned technical problems.

[0006] To achieve the above object, in a first aspect, the present invention provides a high-concurrency transaction data processing method based on edge computing, the method comprising:

[0007] The edge device receives transaction data from multiple user terminals simultaneously through multi-threaded concurrent receiving queues;

[0008] The edge device performs integrity verification and legality verification on the transaction data, and stores the verified transaction data in a local cache;

[0009] The edge device performs sharding processing on the verified transaction data to obtain multiple data shards, and intelligently distributes the multiple data shards to multiple processing modules inside the edge device;

[0010] Each processing module of the edge device processes according to the allocated data slices to obtain a processing result;

[0011] The edge device uses a real-time parallel reduction algorithm to intelligently integrate the processing results obtained by each processing module to obtain transaction response data;

[0012] The edge device compresses the transaction response data using an adaptive compression algorithm, synchronizes the compressed transaction response data to the central server through a multi-encrypted secure transmission channel, and feeds back the synchronization status to the user terminal in real time;

[0013] After confirming that the transaction response data is successfully synchronized to the central server, the edge device safely deletes the verified transaction data stored in the local cache.

[0014] In a second aspect, a high-concurrency transaction data processing system based on edge computing is provided, the system comprising:

[0015] Multiple user terminals;

[0016] At least one edge device, wherein the edge device is configured with:

[0017] A receiving module, used to simultaneously receive transaction data from multiple user terminals through a multi-threaded concurrent receiving queue;

[0018] A verification storage module is used to perform integrity verification and legality verification on the transaction data, and store the verified transaction data in a local cache;

[0019] A data sharding module is used to shard the verified transaction data to generate multiple data shards, and intelligently distribute the data shards to multiple processing modules inside the edge device;

[0020] A parallel processing module is configured with independent computing units for parallel processing according to the allocated data slices to generate processing results;

[0021] The result reduction module uses a real-time parallel reduction algorithm to intelligently integrate the processing results obtained by each processing module to generate transaction response data;

[0022] A data transmission module, used to compress the transaction response data using an adaptive compression algorithm; synchronize the compressed transaction response data to a central server through a multi-encrypted secure transmission channel; and feed back the synchronization status to the user terminal in real time;

[0023] A data cleaning module, after confirming that the transaction response data is successfully synchronized to the central server, safely deletes the verified transaction data stored in the local cache;

[0024] Central server, configured with:

[0025] A data receiving module, used to receive transaction response data synchronized by edge devices;

[0026] A data verification module is used to perform a secondary verification on the received transaction response data;

[0027] A data storage module is used to persistently store transaction response data that has passed the secondary verification;

[0028] A network communication device is used to establish a secure communication link between the user terminal, the edge device and the central server.

[0029] The above technical solution has the following beneficial effects:

[0030] The high-concurrency transaction data processing method based on edge computing proposed in this invention improves the performance and security of the transaction data processing system through innovative technologies such as multi-threaded concurrent reception, intelligent sharding, and parallel processing. The multi-threaded concurrent reception queue can process transaction requests from multiple user terminals at the same time, greatly improving the system throughput; strict integrity and legality verification ensures the reliability of data; data sharding intelligent allocation technology realizes dynamic optimization and load balancing of computing resources; real-time parallel reduction algorithm can quickly integrate processing results and reduce system latency; adaptive compression algorithm and multiple encrypted transmission channels not only optimize network transmission efficiency, but also enhance the security of data transmission; the safe deletion mechanism of transaction data further protects user privacy and system data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. It is obvious that the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 It is a functional block diagram of a high-concurrency transaction data processing system based on edge computing in an embodiment of the present invention;

[0033] Figure 2 It is a flow chart of a high-concurrency transaction data processing method based on edge computing according to an embodiment of the present invention;

[0034] Figure 3is a flowchart of another high-concurrency transaction data processing method based on edge computing according to an embodiment of the present invention;

[0035] Figure 4 It is a functional block diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Embodiment 1

[0038] like Figure 1 As shown, this embodiment provides a high-concurrency transaction data processing system based on edge computing, and the system includes:

[0039] Multiple user terminals;

[0040] At least one edge device, the edge device is configured with: a receiving module, which is used to simultaneously receive transaction data from multiple user terminals through a multi-threaded concurrent receiving queue; a verification storage module, which is used to perform integrity verification and legality verification on the transaction data, and store the verified transaction data in a local cache; a data sharding module, which is used to shard the verified transaction data to generate multiple data shards, and intelligently distribute the data shards to multiple processing modules inside the edge device; a parallel processing module, which is configured with an independent computing unit, which is used to parallelly process the allocated data shards to generate processing results; a result reduction module, which uses a real-time parallel reduction algorithm to intelligently integrate the processing results obtained by each processing module to generate transaction response data; a data transmission module, which is used to compress the transaction response data using an adaptive compression algorithm; synchronize the compressed transaction response data to a central server through a multi-encrypted secure transmission channel; and feed back the synchronization status to the user terminal in real time; a data cleaning module, which securely deletes the verified transaction data stored in the local cache after confirming that the transaction response data has been successfully synchronized to the central server;

[0041] The central server is configured with: a data receiving module for receiving transaction response data synchronized by the edge device; a data verification module for performing secondary verification on the received transaction response data; and a data storage module for persistently storing the transaction response data that has passed the secondary verification;

[0042] A network communication device is used to establish a secure communication link between the user terminal, the edge device and the central server.

[0043] In some embodiments, the receiving module specifically includes: a thread pool management unit, which is used to establish a multi-threaded concurrent receiving queue and configure thread pool parameters; a connection monitoring unit, which is used to monitor the connection request of the user terminal through the network socket, and assign a thread in the thread pool to process each connection request; an identity authentication unit, which is used to authenticate the user terminal that establishes the connection, and create a secure communication channel after the identity authentication is successful; a concurrent receiving unit, which is used to allocate threads to parallelly process transaction data receiving tasks from different user terminals through the configured thread pool, each thread in the thread pool is bound to a user terminal, and receives the data sent by the user terminal in real time; a pre-check unit, which is used to perform a basic integrity and transmission security pre-check on the transaction data received through the secure communication channel, and add the transaction data that passes the pre-check to the multi-threaded concurrent receiving queue.

[0044] In some embodiments, the verification storage module specifically includes: a data extraction unit, which is used to extract the transaction data to be verified from the multi-threaded concurrent receiving queue; a format verification unit, which is used to perform format integrity verification on the extracted transaction data, and verify whether the structure of the transaction data conforms to the expected format through predefined rule matching; a legality verification unit, which is used to perform legality verification on the target field in the transaction data that has passed the format verification, including data type verification, value range verification and logical consistency verification; a status marking unit, which is used to mark the data that has passed the verification as a passed state and the data that has failed the verification as a failed state according to the results of the format integrity verification and the legality verification; a cache storage unit, which is used to write the transaction data marked as a passed state into the local cache and create an index for it; an exception handling unit, which is used to record audit logs for the transaction data marked as a failed state and store them in an isolation area.

[0045] In some embodiments, the data sharding module specifically includes: a sharding rule determination unit, which is used to determine the sharding rules according to the attributes of the transaction data, such as the type, size and processing priority; a data sharding unit, which is used to divide the verified transaction data into multiple data shards according to the sharding rules; a sharding label generation unit, which is used to assign a unique sharding label to each data shard, so as to identify and track the processing status of the data shard; a resource evaluation unit, which is used to dynamically evaluate the current resource occupancy of each processing module inside the edge device to obtain a resource evaluation result; an intelligent allocation unit, which is used to determine the intelligent allocation strategy based on the attributes of the transaction data and the resource evaluation results of each processing module, and allocate multiple data shards to multiple processing modules inside the edge device according to the intelligent allocation strategy.

[0046] In some embodiments, the parallel processing module specifically includes: a shard verification unit, which is used to receive the data shards assigned to the current processing module, and verify the integrity of the data shards and the consistency of their shard labels to ensure that the shard data has not been tampered with or lost; a processing algorithm selection unit, which is used to select an adaptive processing algorithm based on the operation type and content of the data shards; a business processing unit, which is used to call the processing algorithm to perform business logic processing on the data shards, including verifying business rules, performing data operations, and triggering related business operations; a result generation unit, which is used to complete the processing of the data shards and generate processing results, including transaction status, data update information, and intermediate data that can be used for subsequent integration; a result marking unit, which is used to attach timestamps and shard labels to the processing results; a result submission unit, which is used to submit the processing results with attached timestamps and shard labels to the processing result cache area, and notify the real-time parallel reduction module to prepare for integration.

[0047] In some embodiments, the result reduction module specifically includes: a result extraction unit, which is used to extract the processing results generated by each processing module from the processing result cache area, and verify the integrity of each processing result and the consistency of its timestamp and shard label; a sorting unit, which is used to perform preliminary sorting according to the original order of the transaction shards based on the shard labels to ensure the logical consistency of the processing results during the reduction process; a result merging unit, which is used to logically merge and verify multiple shard results of the same transaction based on the shard attributes and business rules of the transaction data to generate an intermediate integrated result; a parallel reduction unit, which is used to call a real-time parallel reduction algorithm, perform parallel processing on multiple intermediate integrated results, and generate transaction response data; a consistency verification unit, which is used to perform consistency verification on the transaction response data; a response cache unit, which is used to mark the transaction response data that has passed the consistency verification as available, and store it in the response cache area for subsequent compression and transmission.

[0048] In some embodiments, the data transmission module specifically includes: a data extraction unit, which is used to extract transaction response data from the response cache area, and perform integrity and consistency checks on the data to ensure that the data is not damaged or tampered with; an adaptive compression unit, which is used to call an adaptive compression algorithm, dynamically adjust the compression ratio according to the size and type of the transaction response data, and compress the transaction response data; an encryption unit, which is used to securely encrypt the compressed transaction response data using multiple encryption algorithms to generate an encrypted data packet; a transmission unit, which is used to send the encrypted data packet to the central server through a pre-established secure transmission channel, and record the status information of the transmission process in real time; a confirmation unit, which is used to receive transmission confirmation information from the central server and verify whether the transaction response data is successfully synchronized to the central server; a status feedback unit, which is used to feedback the synchronization status to the user terminal in real time according to the transmission confirmation information, indicating whether the data synchronization is successful or failed.

[0049] In some embodiments, the data cleaning module specifically includes: a synchronization verification unit, used to confirm that the transaction response data is successfully synchronized to the central server; a cache cleaning unit, used to securely delete the transaction response data in the local cache of the edge device to free up storage space and ensure data privacy.

[0050] In some embodiments, the system also includes: an anomaly monitoring module, which is used to record anomaly logs and store them in an isolation area for transaction data that fails integrity verification or legality verification; when multiple consecutive anomalies are detected or the amount of abnormal data exceeds a preset threshold within a specified time, an alarm mechanism is triggered, and the abnormal status is synchronized to the central server in real time for further analysis and processing.

[0051] In some embodiments, the system further includes: a fault recovery module, which is used to enable a fault recovery mechanism and reload unfinished transaction tasks when processing is interrupted due to a fault in the edge device; transaction response data that has not been synchronized to the central server is resent through the fault recovery mechanism.

[0052] In some embodiments, the data verification module of the central server specifically includes: a signature verification unit, used to verify the validity of the digital signature of the transaction response data; an integrity verification unit, used to check whether the transaction response data has been tampered with during transmission; a consistency check unit, used to compare the consistency of the transaction response data with the original transaction request; a risk assessment unit, used to perform risk level assessment on the transaction response data and identify potential abnormal or suspicious transactions.

[0053] The high-concurrency transaction data processing system based on edge computing proposed in the present invention improves the performance and security of transaction data processing through multi-threaded concurrent reception, intelligent data sharding, parallel processing and real-time parallel reduction. The multi-threaded concurrent reception queue enables the system to process a large number of transaction requests from multiple user terminals at the same time, improving the throughput and response speed of the system; data sharding and intelligent allocation mechanisms ensure the efficient use of computing resources, and maximize the parallel computing capabilities of edge devices through dynamic evaluation and precise allocation; the real-time parallel reduction algorithm realizes the rapid integration of processing results, ensuring the real-time and consistency of transaction data processing; at the same time, multiple encryption and adaptive compression technologies provide security for data transmission, effectively preventing security risks in the data transmission process; the secondary verification mechanism of the central server further enhances the data reliability and security of the system, ensuring the integrity and consistency of transaction data.

[0054] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0055] Embodiment 2

[0056] like Figure 2 As shown, this embodiment provides a high-concurrency transaction data processing method based on edge computing, the method comprising:

[0057] S10: The edge device receives transaction data from multiple user terminals simultaneously through a multi-threaded concurrent receiving queue;

[0058] In this embodiment, the multi-threaded concurrent receiving queue means that the edge device is equipped with a network interface and thread management mechanism with high concurrent processing capabilities. For example, when 100 user terminals initiate transaction requests at the same time, the system can quickly and in parallel receive and cache these transaction data through a pre-set thread pool (e.g., 20-50 working threads). Each thread independently processes one or more connections, which improves the concurrent processing capability of the system and effectively avoids the blocking and delay problems in the traditional single-threaded mode.

[0059] S20: The edge device performs integrity verification and legality verification on the transaction data, and stores the verified transaction data in a local cache;

[0060] Integrity verification verifies whether the data has been tampered with during transmission by calculating the hash value or checksum of the data; legality verification includes checking the format, field integrity, data type, value range, etc. of the transaction data. For example, for financial transactions, the system will verify whether the transaction amount is within a reasonable range, whether the transaction account is valid, and whether the transaction time is within the permitted range. Through these strict checks, illegal or incomplete transaction requests can be effectively filtered out, improving the security and reliability of the system.

[0061] S30: the edge device performs sharding processing on the verified transaction data to obtain multiple data shards, and intelligently distributes the multiple data shards to multiple processing modules inside the edge device;

[0062] The edge device splits the transaction data into multiple logically independent data shards based on its characteristics and size. The intelligent allocation algorithm considers the current load, computing power, and specialization of each processing module and allocates different types of data shards to the most suitable processing module. For example, for complex financial transactions, data from different links such as risk assessment, account verification, and transaction authorization will be allocated to specialized processing modules to achieve parallel and efficient processing.

[0063] S40: Each processing module of the edge device processes according to the allocated data slices to obtain a processing result;

[0064] Each processing module is equipped with dedicated hardware resources and algorithms. For example, the risk assessment module uses machine learning algorithms to analyze the degree of abnormality of transactions in real time, the account verification module calls the identity authentication service, and the transaction authorization module is responsible for checking the compliance of transactions. These modules can be based on GPUs, FPGAs or dedicated AI chips to achieve high-performance parallel computing, greatly improving processing efficiency.

[0065] S50: The edge device intelligently integrates the processing results obtained by each processing module using a real-time parallel reduction algorithm to obtain transaction response data;

[0066] The real-time parallel reduction algorithm is an efficient result integration technology that can quickly merge the calculation results from different processing modules. The algorithm is similar to the reduction operation in MapReduce, but is more lightweight and real-time. For example, for a transaction, the risk assessment module gives the risk level, the account verification module confirms the account validity, and the transaction authorization module checks the compliance. The reduction algorithm will quickly integrate these results according to the preset rules to generate the final transaction response.

[0067] S60: The edge device compresses the transaction response data using an adaptive compression algorithm, synchronizes the compressed transaction response data to the central server through a multi-encrypted secure transmission channel, and feeds back the synchronization status to the user terminal in real time;

[0068] The adaptive compression algorithm dynamically selects the optimal compression scheme based on the characteristics of the transaction response data, minimizing the transmission volume while ensuring data integrity. The multiple encryption channels use multi-layer encryption protocols, such as TLS, IPSec combined with custom encryption algorithms, to ensure the security of data transmission. The real-time feedback of the synchronization status uses WebSocket or similar real-time communication technology, so that the user terminal can instantly understand the progress of transaction processing.

[0069] S70: After confirming that the transaction response data is successfully synchronized to the central server, the edge device safely deletes the verified transaction data stored in the local cache.

[0070] Secure deletion is not just about clearing data, but also about using multiple overwrites, encryption and zeroing to ensure that sensitive transaction data cannot be recovered. This step is an important mechanism to protect user privacy and system security and prevent potential data leakage risks.

[0071] In some embodiments, step S10 specifically includes the following steps:

[0072] S101: Establish a multi-threaded concurrent receiving queue and configure thread pool parameters; the thread pool is used to manage thread resources in the multi-threaded concurrent receiving queue;

[0073] In this embodiment, for example, for high-concurrency transaction scenarios, the edge device can pre-configure a thread pool containing 20-50 worker threads and dynamically adjust the number of threads according to the real-time load. Thread pool parameters include the number of core threads, the maximum number of threads, thread idle time, task queue length, etc. Through fine parameter tuning, the best balance can be achieved between system resource utilization and concurrent processing capabilities, avoiding the performance overhead of thread creation and destruction.

[0074] S102: monitoring the connection request of the user terminal through the network socket, and assigning a thread in the thread pool to each connection request for processing;

[0075] Network sockets are the key communication mechanism connecting user terminals and edge devices. Edge devices use high-performance non-blocking I / O models, such as Netty or epoll, to efficiently monitor and process multiple user terminal connection requests. When a new connection request arrives, the thread pool immediately allocates an idle thread to establish a dedicated network connection channel. For example, in a financial transaction scenario, one thread can simultaneously process different transaction requests from a bank app, web page, and ATM.

[0076] S103: Authenticate the user terminal establishing the connection, and create a secure communication channel after the authentication succeeds;

[0077] Identity authentication is the first line of defense to protect transaction security. This embodiment adopts a multi-factor authentication mechanism, combining multiple authentication methods such as traditional passwords, dynamic tokens, and biometrics. For example, users not only need to enter a password, but also need to perform secondary verification through SMS verification codes, fingerprints, or facial recognition. After successful authentication, the system uses the TLS / SSL protocol to establish an end-to-end encrypted communication channel to ensure the security and integrity of subsequent transaction data transmission.

[0078] S104: Allocate threads to process transaction data receiving tasks from different user terminals in parallel through the configured thread pool. Each thread in the thread pool is bound to a user terminal to receive data sent by the user terminal in real time.

[0079] Each thread is responsible for receiving data from a user terminal, which can achieve true parallel processing. For example, when 50 users initiate transaction requests at the same time, 50 independent threads can receive and process these requests at the same time, improving the throughput of the system. Each thread is equipped with an independent data buffer and processing logic to minimize competition and blocking between threads.

[0080] S105: Perform a basic integrity and transmission security pre-check on the transaction data received through the secure communication channel, and add the transaction data that passes the pre-check to a multi-threaded concurrent receiving queue.

[0081] Pre-check is a lightweight security filter performed before formal data verification. The system will quickly check the basic integrity of the data packet, such as data length, basic format, transmission protocol consistency, etc. At the same time, it will perform transmission security checks, including detecting whether the data packet has tampering, replay attacks, and other anomalies. Only data that passes these pre-checks will be added to the multi-threaded concurrent receiving queue, effectively blocking most potential security threats.

[0082] The advantages of this solution are: through the multi-threaded concurrent receiving architecture, the system's ability to handle high-concurrency transaction requests is improved, and the transaction needs of a large number of users can be processed at the same time, greatly improving the user experience; the multi-level security authentication and pre-check mechanism embeds strict security control in every link from connection establishment to data reception, effectively protecting the security of user transaction data; the dynamic management of the thread pool and the precise binding of threads to user terminals realize the efficient utilization and optimal allocation of computing resources, and reduce system resource consumption; the solution has good scalability and adaptability, and can flexibly adjust the thread pool parameters and processing strategies according to different business scenarios and load requirements.

[0083] In some embodiments, step S20 specifically includes the following steps:

[0084] S201: extracting transaction data to be verified from the multi-thread concurrent receiving queue;

[0085] In this embodiment, data extraction is an efficient non-blocking operation. The system uses a lock-free queue (such as ConcurrentLinkedQueue) to achieve fast data extraction and ensure high performance in a multi-threaded environment. For example, in a financial transaction scenario, the system can extract hundreds or even thousands of transaction data from the concurrent receiving queue per second and pass it to the subsequent verification module. The extraction process adopts a batch reading strategy, which can obtain multiple pieces of data to be verified at one time, further improving the system processing efficiency.

[0086] S202: Perform format integrity check on the extracted transaction data, and verify whether the structure of the transaction data conforms to the expected format through predefined rule matching. If the transaction data does not conform to the expected format, it is directly marked as a failed state;

[0087] The system predefines strict data format templates, including required fields, field order, separators, etc. For example, for financial transactions, transaction data is required to contain fixed fields such as transaction type, transaction amount, account ID, timestamp, etc., and each field has specific format requirements. If any key field is missing or the format does not match, the system will immediately mark the transaction data as failed to prevent non-compliant data from entering the subsequent processing flow.

[0088] S203: Performing a validity check on the target field in the transaction data that has passed the format check, which includes the following: data type check, used to verify whether the data type of the target field conforms to the preset data type; value range check, used to verify whether the value field is within the preset range; logical consistency check, used to verify whether the logical relationship between the target fields is reasonable;

[0089] Data type verification ensures that the data type of each field matches accurately, for example, the transaction amount must be a numeric type and the account ID must be a string. Value range verification prevents abnormal data, for example, the transaction amount must be between 0 and 1,000,000, and the transaction time must be within a certain range before and after the current system time. Logical consistency verification is more complex and requires analyzing the relationship between multiple fields. For example, for cross-border transactions, it is necessary to verify the logical consistency between the transaction currency, exchange rate, and actual amount to prevent fraud.

[0090] S204: According to the results of the format integrity check and the legality check, the data that has passed the check is marked as a passed state, and the data that has failed the check is marked as a failed state;

[0091] The system uses enumerations or status codes to mark the verification results of each transaction data, such as "PASS", "FAIL_FORMAT", "FAIL_TYPE", "FAIL_RANGE", etc. This fine-grained status marking not only facilitates subsequent processing, but also provides detailed information for auditing and security analysis.

[0092] S205: Write the transaction data marked as passed into the local cache and create an index for it;

[0093] The local cache uses high-performance key-value storage technology, such as Redis or local memory cache. Each verified transaction data will be assigned a unique transaction ID as the primary key, and a multi-dimensional index will be established, such as by time, account, transaction type, etc. This indexing mechanism makes subsequent data processing and retrieval fast and efficient. For example, all transaction records of a specific time period and a specific account can be quickly located at the millisecond level.

[0094] S206: Record an audit log for the transaction data marked as failed and store it in the isolation area.

[0095] Audit log records include complete details of data verification failures, such as failure reasons, data content, source IP, timestamp, etc. The quarantine area is a secure storage space for storing data that has failed verification, which is convenient for subsequent security analysis and evidence collection. The system will regularly conduct risk assessments on quarantine area data and trigger security alerts when necessary.

[0096] The advantages of this solution are: through a multi-level, fine-grained data verification mechanism, it fundamentally prevents non-compliant and unsafe transaction data from entering the core processing flow of the system, thereby improving the security and reliability of the system; flexible data status marking and multi-dimensional index design not only facilitate real-time data processing, but also provide a solid data foundation for subsequent big data analysis and risk control; the design of audit logs and isolation areas reflects the system's comprehensive consideration of data security, and provides strong support for security traceability and risk management; the entire verification process adopts a high-performance, non-blocking design, which can maintain extremely low latency and high throughput even in the face of high-concurrency scenarios with thousands of transactions per second.

[0097] In some embodiments, step S30 specifically includes the following steps:

[0098] S301: Determine sharding rules based on attributes of transaction data including type, size, and processing priority;

[0099] In this embodiment, the sharding rule is a dynamic and intelligent decision-making process. The system formulates fine-grained sharding strategies based on the multi-dimensional attributes of transaction data. For example, for financial transaction scenarios, different sharding granularities can be formulated according to transaction type (such as payment, transfer, financial management), transaction amount (small, medium, large), customer level (ordinary, VIP), etc. Small-amount, high-frequency transactions use finer sharding, while large-amount, complex transactions require larger sharding units to ensure that each shard can be fully and accurately processed.

[0100] S302: Divide the verified transaction data into multiple data shards according to the sharding rule;

[0101] The system can use intelligent sharding algorithms to divide transaction data into multiple logical blocks according to predefined rules. For example, a complex cross-border transaction can be decomposed into multiple independent data shards such as exchange rate calculation, fund verification, and compliance check. The granularity and method of sharding can be dynamically adjusted to adapt to transaction data of different types and complexities.

[0102] S303: assigning a unique shard label to each data shard to identify and track the processing status of the data shard;

[0103] Each shard label contains information such as a globally unique identifier (GUID), the original transaction ID, the shard sequence number, the creation timestamp, etc. For example, a complete shard label is similar to: "TXN-20241201-123456-SHARD-001". This design not only facilitates real-time tracking of the processing status of each shard, but also provides a reliable tracking mechanism for subsequent data reorganization, auditing, and fault recovery.

[0104] S304: Dynamically evaluate the current resource occupancy of each processing module in the edge device to obtain a resource evaluation result;

[0105] The system continuously monitors key indicators such as CPU utilization, memory usage, and processing queue length of each processing module. For example, for edge devices with multiple professional processing modules (such as risk control modules, encryption modules, and routing modules), the system can collect resource usage of each module in real time and build a dynamic resource usage profile. This real-time, fine-grained resource monitoring enables the system to make optimal allocation decisions.

[0106] S305: Determine an intelligent allocation strategy based on the attributes of the transaction data and the resource evaluation results of each processing module;

[0107] Transaction data attributes are a multi-dimensional information set that describes transaction characteristics. These attributes comprehensively cover the essential characteristics of transactions, including transaction type, financial scenario, transaction amount level, risk characteristics, time dimension, and technical attributes. By carefully describing these attributes, the system can construct a multi-dimensional transaction feature vector, providing rich and accurate input information for subsequent intelligent allocation.

[0108] Transaction data attributes can be specifically divided into four main dimensions: transaction type attributes, risk attributes, time attributes, and technical attributes. In terms of transaction types, the system considers different transaction scenarios such as payment, transfer, and recharge; in risk attributes, risk profiles are established by evaluating the risk level, abnormal characteristics, and account credibility of transactions; time attributes focus on the time characteristics and frequency of transactions; technical attributes include key indicators such as data complexity, encryption level, and data size. This multi-dimensional attribute description provides a comprehensive and detailed feature foundation for subsequent intelligent allocation.

[0109] The professionalism of the processing module is accurately characterized by quantitative indicators, mainly including three dimensions: professional ability index, performance index and learning ability. The professional ability index reflects the processing ability of the module in a specific transaction scenario; the performance index focuses on the processing efficiency and resource utilization of the module; and the learning ability evaluates the module's adaptability to new scenarios and algorithm iteration capabilities.

[0110] The intelligent allocation strategy is a dynamic matching algorithm based on machine learning. The algorithm takes the transaction data attribute vector and the processing module resource status as input, and achieves the optimal mapping of transactions to processing modules through steps such as feature standardization, multi-dimensional matching scoring, and dynamic weight adjustment. For example, for high-risk large-value cross-border transactions, the system will give priority to professional modules with complex risk control capabilities; while for standard small-value payment transactions, they can be allocated to general modules with stronger processing capabilities.

[0111] By introducing reinforcement learning, clustering algorithms and prediction models, the system can dynamically adjust the allocation strategy based on real-time feedback of allocation results. Reinforcement learning enables the system to learn from historical allocation experience; clustering algorithms help identify transaction patterns and module matching features; and prediction models predict the best allocation plan based on historical data. This multi-algorithm collaborative learning mechanism enables the allocation strategy to have the ability to adapt and continuously optimize, and can quickly respond to changing business scenarios and technical environments.

[0112] S306: Allocate the multiple data shards to the multiple processing modules inside the edge device according to the intelligent allocation strategy.

[0113] Shard allocation is a precise and efficient process. The system uses a lightweight scheduler that can complete intelligent routing of data shards at the millisecond level. The allocation process not only considers the current resource status, but also predicts the processing capacity and response time of each module. For example, for a transaction system with a large concurrency, the shard allocation can be adjusted dynamically to ensure that each processing module can obtain a balanced workload and avoid single-point performance bottlenecks.

[0114] The advantages of this solution are: through multi-dimensional, intelligent data sharding and allocation mechanism, the parallel capability and system throughput of edge devices in processing complex transactions are improved, and it can cope with high-concurrency, heterogeneous transaction scenarios; dynamic resource evaluation and intelligent allocation strategy enable the system to balance computing resources in real time and accurately, maximize system performance, and reduce resource waste; unique shard labels and fine-grained tracking mechanisms not only provide data traceability, but also provide support for system security auditing and fault recovery; the solution is highly scalable and adaptable, and can flexibly adjust sharding and allocation strategies according to different business scenarios and load characteristics.

[0115] In some embodiments, step S40 specifically includes the following steps:

[0116] S401: Receive the data shards assigned to the current processing module, and verify the integrity of the data shards and the consistency of their shard labels to ensure that the shard data has not been tampered with or lost;

[0117] In this embodiment, the system uses multiple verification mechanisms, including hash verification, digital signatures, and blockchain technology. For example, for each data shard, the system calculates and stores the SHA-256 hash value of its content and compares it with the original hash value in the shard tag. At the same time, the digital signature of the shard is verified through public key encryption technology to ensure that the data has not been tampered with during transmission. If any inconsistency is found, the system will immediately refuse to process the shard and trigger a security alarm mechanism.

[0118] S402: Selecting an adaptive processing algorithm based on the operation type and content of the data shard;

[0119] The system has established a flexible algorithm mapping library to dynamically select the most suitable processing algorithm based on the characteristics of data sharding. For example, for financial transaction scenarios, different types of transactions (such as payment, transfer, and financial management) will match different exclusive processing algorithms. The system also supports dynamic loading of machine learning algorithms, and can intelligently adjust processing strategies based on real-time data characteristics to achieve more accurate business logic processing.

[0120] S403: calling the processing algorithm to perform business logic processing on the data shards, which includes verifying business rules, performing data operations, and triggering related business operations;

[0121] The processing process is not just a simple data conversion, but a complex multi-step verification and calculation process. For example, for a cross-border payment transaction, the processing algorithm may include: compliance check (anti-money laundering, sanctions list), exchange rate calculation, real-time risk control assessment, account balance verification, tax calculation and other sub-steps. Each step has strict business rules verification to ensure the legality and accuracy of the transaction.

[0122] S404: Complete the processing of data shards and generate processing results, which include transaction status, data update information, and intermediate data that can be used for subsequent integration;

[0123] The results generated by the system include not only the final status of the transaction (such as success, failure, pending confirmation), but also a wealth of intermediate processing data. For example, for a transaction, the processing result includes: transaction unique identifier, processing time, account information involved, actual transaction amount, exchange rate information, risk control score, etc.

[0124] S405: Add a timestamp and a fragmentation tag to the processing result to obtain the processing result with the added timestamp and fragmentation tag;

[0125] The system adds a timestamp accurate to microseconds to each processing result to record the exact time point of the processing. The shard label retains all the context information of the original data shard. For example, a complete processing result label is: "TXN-20241201-123456-SHARD-001-PROCESSED-2024-12-01T12:34:56.789Z". This design not only facilitates accurate tracking, but also provides a reliable basis for subsequent data auditing and replay.

[0126] S406: Submit the processing result with the timestamp and the shard tag attached to the processing result cache area, and notify the real-time parallel reduction module to prepare for integration.

[0127] The system uses a high-performance distributed cache (such as Redis) as a temporary storage area for processing results. Each processing result is accurately stored and indexed, waiting for subsequent reduction processing. The notification mechanism adopts a publish-subscribe mode, and the real-time parallel reduction module can immediately know the new processing results and prepare for data integration. This design ensures the real-time and efficient data processing.

[0128] The advantages of this solution are: through multi-level data integrity verification and intelligent algorithm selection, the security and accuracy of system processing are improved, and it can cope with complex and changing business scenarios; fine-grained processing result design and rich intermediate data not only meet immediate business needs, but also provide a data basis for subsequent big data analysis and intelligent decision-making; precise timestamp and shard label mechanism realizes the complete tracking and verifiability of the data processing process, and provides support for system auditing and compliance; flexible caching and notification mechanism ensures the real-time and efficiency of data processing, and has strong scalability and adaptability.

[0129] In some embodiments, step S50 specifically includes the following steps:

[0130] S501: extracting the processing results generated by each processing module from the processing result cache area, and verifying the integrity of each processing result and the consistency of its timestamp and sharding tag;

[0131] In this embodiment, the system uses a multiple verification mechanism to check not only the physical integrity of the data, but also the logical consistency. For example, for a complex financial transaction, the system will compare the timestamps of each shard result to ensure the rationality of the processing order. At the same time, the integrity of each processing result is verified through a cryptographic hash algorithm, and any minor data tampering will be immediately detected and rejected. The verification process also includes cross-referencing shard tags to ensure that all results belong to different processing stages of the same transaction.

[0132] S502: Perform preliminary sorting according to the original order of the transaction shards based on the shard labels to ensure the logical consistency of the processing results during the reduction process;

[0133] The system has designed an intelligent sorting algorithm that is not only based on timestamps, but also considers the sequence information in the shard tags. For example, for a cross-border payment transaction, there are multiple shards including risk control assessment, exchange rate calculation, and fund verification. The sorting algorithm will accurately restore the original order of transaction processing according to the predefined business process logic. This approach ensures that even in the case of parallel processing, the final data integration still follows a strict logical order.

[0134] S503: Based on the sharding attributes and business rules of the transaction data, multiple sharding results of the same transaction are logically merged and verified to generate an intermediate integration result;

[0135] The system uses a complex rule engine and intelligent merging algorithm, which not only simply splices data, but also performs in-depth logical verification. For example, for an international trade payment, the system needs to merge the processing results from different modules such as risk control, compliance, finance, and taxation. The merging process will check the consistency of the results of each shard, resolve conflicts, and generate a comprehensive intermediate integration result. This process is similar to a jigsaw puzzle, ensuring that the final result is comprehensive and accurate.

[0136] S504: calling a real-time parallel reduction algorithm to process multiple intermediate integration results in parallel and generate transaction response data;

[0137] The system uses a real-time parallel reduction algorithm that can process multiple intermediate integration results at the same time. For example, for large-scale financial transaction systems, it can process the intermediate results of hundreds or even thousands of transactions at the same time. The reduction algorithm not only focuses on computational efficiency, but also ensures the accuracy of data processing. It can dynamically adjust computing resources and flexibly allocate processing power according to the complexity of different transactions to achieve efficient and accurate data integration.

[0138] The Real-time Parallel Reduction Algorithm is a distributed computing algorithm that is specifically designed to handle large-scale, high-concurrency data computing scenarios. Its goal is to quickly and efficiently aggregate the scattered intermediate computing results into a single, accurate final result. The uniqueness of the algorithm lies in its parallel processing capability, which can process multiple data streams simultaneously and complete complex data reduction operations in milliseconds. This technology is not only a computing method, but also an intelligent data processing paradigm that can dynamically adjust computing resources and flexibly allocate processing power according to the complexity of the task.

[0139] The working principle of the algorithm can be summarized into three stages: data sharding and parallel processing, intermediate result aggregation, and dynamic resource scheduling. In the data sharding stage, the algorithm intelligently splits the huge data set into multiple relatively independent subsets, each of which can be processed in parallel on distributed computing nodes. This sharding strategy ensures a high degree of parallelism in computing, allowing the system to process hundreds or even thousands of data fragments at the same time. The intermediate result aggregation adopts a tree reduction strategy, which gradually integrates the scattered computing results into the final unified result through multi-level step-by-step merging. The dynamic resource scheduling mechanism can evaluate the load of each computing node in real time, and dynamically allocate and balance computing resources according to the complexity of the task and the real-time status of system resources.

[0140] S505: Perform consistency verification on the transaction response data;

[0141] The system is designed with a multi-level verification mechanism, including mathematical model verification, business rule checking, and machine learning anomaly detection. For example, for financial transactions, the verification process will check the rationality of the transaction amount, the consistency of the account balance, the accuracy of the tax calculation, etc. The system will also perform cross-module cross-verification to ensure that there are no logical contradictions in the data generated by different processing links. This comprehensive verification method improves the reliability of the system output data.

[0142] S506: Mark the transaction response data that has passed the consistency verification as available, and store it in the response cache area for subsequent compression and transmission.

[0143] The system uses high-performance distributed caching technology to assign a unique identifier to each verified transaction response data. The storage process not only considers the persistence of data, but also optimizes subsequent data compression and transmission. For example, the system can select the most suitable compression algorithm based on the characteristics of the data to reduce the storage space and transmission bandwidth occupied. At the same time, the cache area supports fine-grained access control and audit tracking, providing a strong guarantee for data security and compliance.

[0144] The advantages of this solution are: through multi-level, intelligent data verification and merging mechanisms, the accuracy and reliability of the system in processing complex transactions are improved, and it can cope with highly dynamic and complex business scenarios; advanced parallel reduction algorithms and distributed computing technologies have greatly improved the processing performance of the system, and it can handle a large number of complex transactions at the same time and has scalability; sophisticated consistency verification and intelligent sorting mechanisms not only ensure the logic of data processing, but also provide a high-quality data foundation for subsequent data analysis and decision-making; flexible caching and storage design realizes efficient data compression, secure transmission and traceability.

[0145] In some embodiments, step S60 specifically includes the following steps:

[0146] S601: extracting transaction response data from the response cache area, and performing integrity and consistency checks on the data to ensure that the data is not damaged or tampered with;

[0147] In this embodiment, the system uses multiple verification mechanisms, including hash value comparison, digital signature verification, and blockchain technology. For example, for the response data of a financial transaction, the system first calculates the SHA-256 hash value of the data and compares it with the stored original hash value. At the same time, the digital signature of the data is verified using the public key infrastructure (PKI) to ensure that the data has not been tampered with during transmission. This multi-level verification method can quickly identify any data integrity issues and provide a reliable data foundation for subsequent processing.

[0148] S602: calling an adaptive compression algorithm, dynamically adjusting the compression ratio according to the size and type of the transaction response data, and compressing the transaction response data;

[0149] The system is designed with an intelligent compression algorithm that can dynamically select the best compression strategy based on the characteristics of the data. For example, for different types of transaction response data (such as text, images, and videos), the system will automatically select the most suitable compression algorithm, such as Huffman coding for text and JPEG compression for images. The compression algorithm also considers the sensitivity and importance of the data, minimizing the data volume while ensuring data quality. This intelligent compression method not only saves storage space and transmission bandwidth, but also improves the overall performance of the system.

[0150] S603: securely encrypt the compressed transaction response data using multiple encryption algorithms to generate an encrypted data packet;

[0151] The system uses a layered encryption strategy that combines symmetric and asymmetric encryption techniques. For example, data is first symmetrically encrypted using AES-256, and then the AES key is encrypted using the RSA-2048 asymmetric encryption algorithm. This multiple encryption method provides multiple layers of security, and even if one layer of encryption is compromised, the data can still be protected. The encryption process also dynamically generates a one-time key, which further increases the difficulty of cracking. For sensitive financial transaction data, this high-intensity encryption scheme can effectively prevent data leakage and unauthorized access.

[0152] S604: Send the encrypted data packet to the central server through a pre-established secure transmission channel, and record the status information of the transmission process in real time;

[0153] The system uses a secure transmission protocol based on TLS1.3 to establish an end-to-end encrypted communication channel. The transmission process uses dynamic routing and multi-path transmission technology to automatically select the safest and most efficient network path. For example, for cross-border financial transactions, the system can monitor network conditions in real time and dynamically adjust transmission routes. At the same time, each key node in the transmission process will record detailed status information, including transmission time, network delay, data packet integrity, etc., to provide important data for subsequent audits and optimizations.

[0154] S605: Receive transmission confirmation information from the central server to verify whether the transaction response data is successfully synchronized to the central server;

[0155] The system is designed with an intelligent confirmation mechanism that not only verifies whether the data is successfully transmitted, but also checks the integrity and consistency of the data. For example, the central server will return a confirmation message containing the data hash value, transmission timestamp, and checksum. The edge device will perform multiple verifications on this information to ensure that the data has been completely and accurately synchronized to the central server. If any abnormality is found, the system will automatically trigger a retransmission mechanism to ensure reliable synchronization of data.

[0156] S606: Feedback the synchronization status to the user terminal in real time according to the transmission confirmation information, indicating whether the data synchronization is successful or failed;

[0157] The system is designed with an intelligent notification mechanism that can generate personalized user prompts based on different synchronization statuses. For example, for successfully synchronized transactions, the system will send a concise success notification; for synchronization failures, detailed error information and suggested solutions will be provided. Notification methods support multiple channels, such as SMS, in-app notifications, emails, etc., to ensure that users can keep up to date with the latest status of data synchronization.

[0158] S607: After confirming that the transaction response data is successfully synchronized to the central server, securely delete the transaction response data in the local cache of the edge device to free up storage space and ensure data privacy.

[0159] The system uses multiple data clearing technologies to ensure that sensitive data in the local cache is completely and irreversibly deleted. For example, using multiple overwrite technology, the original data is overwritten with random data multiple times to completely eliminate data traces. The deletion process also records logs to provide a basis for subsequent compliance audits. This secure deletion method not only frees up storage space on edge devices, but also maximizes the protection of user data privacy.

[0160] The advantages of this solution are: through multi-level data verification, adaptive compression and multiple encryption technologies, a comprehensive data security protection system is built to effectively protect the integrity and confidentiality of sensitive transaction data; the intelligent transmission and synchronization mechanism, combined with dynamic routing and multi-path transmission technology, improves the reliability and efficiency of data transmission, and can ensure accurate data synchronization even in complex network environments; real-time status feedback and personalized notification mechanisms improve the user experience, allowing users to understand the status of data synchronization in real time; secure deletion technology not only protects data privacy, but also optimizes the storage resources of edge devices, and is secure, efficient and user-friendly.

[0161] like Figure 3 As shown, in some embodiments, the method further includes:

[0162] S80: The edge device records an exception log and stores it in an isolation area for transaction data that fails the integrity check or the legality check. When multiple consecutive exceptions are detected or the amount of abnormal data exceeds a preset threshold within a specified time, an alarm mechanism is triggered and the abnormal status is synchronized to the central server in real time for further analysis and processing.

[0163] In this embodiment, the edge device first establishes a dedicated data isolation area to store transaction data that has not passed the integrity and legality verification. This isolation area uses strict access control and encrypted storage technology to ensure that abnormal data will not affect the normal operation of the system. For example, for a financial transaction data, if its hash value verification fails or the digital signature verification fails, the system will immediately move the data to the isolation area and generate a detailed exception log.

[0164] The exception log records not only the basic information of the exception data, but also detailed information such as the timestamp of the exception, the exception type, the data source, the preliminary diagnosis, etc. The system uses a structured log record format to facilitate subsequent data analysis and security audits. For example, for a detected data integrity exception, the log contains: the time of the exception, the IP address of the data source, the hash value of the original data, the expected hash value, the exception type (such as data tampering, transmission damage), and other key information.

[0165] The system is designed with an intelligent anomaly threshold monitoring algorithm that can dynamically analyze the frequency and pattern of abnormal data. The alarm mechanism will be triggered when any of the following conditions are detected: the number of abnormal data exceeds 5% of the total transaction data within 5 consecutive minutes; the cumulative amount of abnormal data exceeds 100 within an hour; abnormal data from the same data source is detected 3 times in a row. After the alarm is triggered, the system will generate high-priority alarm information and synchronize it to the central server in real time through multiple channels.

[0166] The system uses an efficient real-time communication protocol to ensure that the alarm information can be transmitted to the central server quickly and reliably. The alarm information not only contains a detailed description of the abnormal data, but also carries a preliminary risk assessment and recommended processing strategy. For example, for a detected suspicious data source, the alarm information includes: a recommended IP address blocking strategy, a preliminary threat level assessment, and recommended further investigation directions. After receiving the alarm information, the central server can immediately start the security response process, such as triggering an intrusion detection system, starting a security audit program, or performing preventive network isolation measures.

[0167] In order to enhance the system's adaptability and learning capabilities, the abnormal data processing mechanism also integrates machine learning technology. The system continuously analyzes abnormal logs to identify potential attack patterns and system vulnerabilities. Through in-depth analysis of historical abnormal data, the system can continuously optimize the anomaly detection algorithm and improve the ability to identify and defend against new security threats. For example, the system can use anomaly detection algorithms such as Isolation Forest or Local Outlier Factor (LOF) to automatically learn and identify characteristic patterns of data anomalies.

[0168] The advantages of this solution are: by establishing a dedicated data isolation area and a detailed exception log recording mechanism, the system can comprehensively and accurately capture and manage potential security threats, and effectively prevent the negative impact of abnormal data on the system; the intelligent anomaly detection and alarm mechanism, combined with dynamic threshold monitoring and multi-channel alarm synchronization, can achieve rapid response and active defense to security threats, and improve the security resilience of the system; by integrating machine learning technology, the system has the ability of continuous learning and self-optimization, and can continuously improve the level of identification and defense of new security threats.

[0169] In some embodiments, the method further comprises:

[0170] S90: When the edge device is interrupted due to a fault, the fault recovery mechanism is enabled to reload the unfinished transaction tasks; the transaction response data that has not been synchronized to the central server is resent through the fault recovery mechanism.

[0171] In this embodiment, the system designs a multi-level fault detection and recovery framework that can quickly identify and handle various types of faults in edge devices. The fault detection module continuously monitors the operating status of the device, including key indicators such as hardware performance, system resources, and network connections. When an abnormal situation that affects the normal operation of the system is detected, such as excessive CPU load, memory exhaustion, and network connection interruption, the system will immediately trigger the fault recovery process.

[0172] The system uses real-time task status snapshot technology to regularly record the execution progress, key parameters and intermediate status of each transaction task. This snapshot mechanism not only records the basic information of the task, but also includes detailed execution context. For example, for a complex financial transaction, the system will record key information such as the initial parameters of the transaction, completed processing steps, unfinished subtasks, and the current processing stage.

[0173] When an edge device detects a fault, the system will accurately locate the interrupted transaction task based on the task status snapshot and automatically start the reload process. The reload process includes the following steps: restore the task execution environment from the most recent valid state snapshot; reinitialize the unfinished transaction task; automatically adjust the task execution parameters to adapt to the current system status; and continue the unfinished transaction processing from the interruption point. For example, for an interrupted cross-border payment transaction, the system can accurately restart from the last confirmed processing step to ensure the continuity and accuracy of the transaction.

[0174] The system has designed an intelligent data retransmission strategy that can reliably handle transaction response data that has not been successfully synchronized to the central server. The retransmission mechanism includes the following core functions: establishing a priority queue for unsynchronized data; automatically detecting and filtering data to be retransmitted; and dynamically adjusting the retransmission strategy to avoid network congestion and repeated transmission. For example, for a batch of transaction response data that has not been successfully synchronized, the system will intelligently formulate the best retransmission order and retransmission plan based on the timestamp, importance, and sensitivity of the data.

[0175] In order to improve the reliability of fault recovery, the system also integrates multiple fault-tolerant technologies. This includes: distributed backup mechanism, redundant storage of key task status among multiple edge devices; blockchain-based state synchronization technology to ensure the immutability of state snapshots; adaptive network transmission protocol, which can optimize data retransmission under different network conditions. For example, when network bandwidth is limited, the system can dynamically adjust the data compression rate and transmission strategy to ensure that key data can be reliably retransmitted.

[0176] The advantages of this solution are: through sophisticated task status snapshots and intelligent fault recovery mechanisms, the system can quickly and accurately restore transaction processing when a fault occurs, minimizing the impact of the fault on business continuity; multi-level fault detection and reload technology, combined with intelligent data retransmission strategies, provides a comprehensive and reliable fault recovery solution that can cope with various complex system failure scenarios; by integrating distributed backup, blockchain synchronization and adaptive transmission technology, the system has fault tolerance and data recovery performance; this comprehensive fault recovery solution not only ensures system reliability and data integrity, but also provides a scalable and highly resilient technical solution for complex edge computing environments.

[0177] The present application also provides a machine-readable storage medium on which a program is stored, and the program implements the above method when executed by a processor. The present application also provides a computer program product, including a computer program, and the computer program implements the above method when executed by a processor.

[0178] The embodiment of the present application also provides a computer device, which may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor A01, a network interface A02, a display screen A04, an input device A05 and a memory (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A06. The non-volatile storage medium A06 stores an operating system B01 and a computer program B02. The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A06. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor A01, a method for a decision engine is implemented. The display screen A04 of the computer device can be a liquid crystal display or an electronic ink display, and the input device A05 of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0179] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0181] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0184] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory. Memory includes non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0185] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0186] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0187] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A high-concurrency transaction data processing method based on edge computing, characterized in that: The method comprises: S10: The edge device receives transaction data from multiple user terminals simultaneously through a multi-threaded concurrent receiving queue; S20: The edge device performs integrity check and legality check on the transaction data, and stores the verified transaction data in a local cache; S30: the edge device performs sharding processing on the verified transaction data to obtain multiple data shards, and intelligently distributes the multiple data shards to multiple processing modules inside the edge device; S40: Each processing module of the edge device processes according to the allocated data slices to obtain a processing result; S50: The edge device intelligently integrates the processing results obtained by each processing module using a real-time parallel reduction algorithm to obtain transaction response data; S60: The edge device compresses the transaction response data using an adaptive compression algorithm, synchronizes the compressed transaction response data to the central server through a multi-encrypted secure transmission channel, and feeds back the synchronization status to the user terminal in real time; S70: After confirming that the transaction response data is successfully synchronized to the central server, the edge device safely deletes the verified transaction data stored in the local cache.

2. The method according to claim 1, characterized in that Step S10 specifically includes the following steps: S101: Establish a multi-threaded concurrent receiving queue and configure thread pool parameters; the thread pool is used to manage thread resources in the multi-threaded concurrent receiving queue; S102: monitoring the connection request of the user terminal through the network socket, and assigning a thread in the thread pool to each connection request for processing; S103: Authenticate the user terminal establishing the connection, and create a secure communication channel after the authentication succeeds; S104: Allocate threads to process transaction data receiving tasks from different user terminals in parallel through the configured thread pool. Each thread in the thread pool is bound to a user terminal to receive data sent by the user terminal in real time. S105: Perform a basic integrity and transmission security pre-check on the transaction data received through the secure communication channel, and add the transaction data that passes the pre-check to a multi-threaded concurrent receiving queue.

3. The method according to claim 1, characterized in that Step S20 specifically includes the following steps: S201: extracting transaction data to be verified from the multi-thread concurrent receiving queue; S202: Perform format integrity check on the extracted transaction data, and verify whether the structure of the transaction data conforms to the expected format through predefined rule matching. If the transaction data does not conform to the expected format, it is directly marked as a failed state; S203: Performing a validity check on the target field in the transaction data that has passed the format check, which includes the following: data type check, used to verify whether the data type of the target field conforms to the preset data type; value range check, used to verify whether the value field is within the preset range; logical consistency check, used to verify whether the logical relationship between the target fields is reasonable; S204: According to the results of the format integrity check and the legality check, the data that has passed the check is marked as a passed state, and the data that has failed the check is marked as a failed state; S205: Write the transaction data marked as passed into the local cache and create an index for it; S206: Record an audit log for the transaction data marked as failed and store it in the isolation area.

4. The method according to claim 1, characterized in that: Step S30 specifically includes the following steps: S301: Determine sharding rules based on attributes of transaction data including type, size, and processing priority; S302: Divide the verified transaction data into multiple data shards according to the sharding rule; S303: assigning a unique shard label to each data shard to identify and track the processing status of the data shard; S304: Dynamically evaluate the current resource occupancy of each processing module in the edge device to obtain a resource evaluation result; S305: Determine an intelligent allocation strategy based on the attributes of the transaction data and the resource evaluation results of each processing module; S306: Allocate the multiple data shards to the multiple processing modules inside the edge device according to the intelligent allocation strategy.

5. The method according to claim 1, characterized in that Step S40 specifically includes the following steps: S401: Receive the data shards assigned to the current processing module, and verify the integrity of the data shards and the consistency of their shard labels to ensure that the shard data has not been tampered with or lost; S402: Selecting an adaptive processing algorithm based on the operation type and content of the data shard; S403: calling the processing algorithm to perform business logic processing on the data shards, which includes verifying business rules, performing data operations, and triggering related business operations; S404: Complete the processing of data shards and generate processing results, which include transaction status, data update information, and intermediate data that can be used for subsequent integration; S405: Add a timestamp and a fragmentation tag to the processing result to obtain the processing result with the added timestamp and fragmentation tag; S406: Submit the processing result with the timestamp and the shard tag attached to the processing result cache area, and notify the real-time parallel reduction module to prepare for integration.

6. The method according to claim 1, characterized in that Step S50 specifically includes the following steps: S501: extracting the processing results generated by each processing module from the processing result cache area, and verifying the integrity of each processing result and the consistency of its timestamp and sharding tag; S502: Perform preliminary sorting according to the original order of the transaction shards based on the shard labels to ensure the logical consistency of the processing results during the reduction process; S503: Based on the sharding attributes and business rules of the transaction data, multiple sharding results of the same transaction are logically merged and verified to generate an intermediate integration result; S504: calling a real-time parallel reduction algorithm to process multiple intermediate integration results in parallel and generate transaction response data; S505: Perform consistency verification on the transaction response data; S506: Mark the transaction response data that has passed the consistency verification as available, and store it in the response cache area for subsequent compression and transmission.

7. The method according to claim 1, characterized in that Step S60 specifically includes the following steps: S601: extracting transaction response data from the response cache area, and performing integrity and consistency checks on the data to ensure that the data is not damaged or tampered with; S602: calling an adaptive compression algorithm, dynamically adjusting the compression ratio according to the size and type of the transaction response data, and compressing the transaction response data; S603: securely encrypt the compressed transaction response data using multiple encryption algorithms to generate an encrypted data packet; S604: Send the encrypted data packet to the central server through a pre-established secure transmission channel, and record the status information of the transmission process in real time; S605: Receive transmission confirmation information from the central server to verify whether the transaction response data is successfully synchronized to the central server; S606: Feedback the synchronization status to the user terminal in real time according to the transmission confirmation information, indicating whether the data synchronization is successful or failed; S607: After confirming that the transaction response data is successfully synchronized to the central server, securely delete the transaction response data in the local cache of the edge device to free up storage space and ensure data privacy.

8. The method according to claim 1, characterized in that Also includes: S80: The edge device records an exception log and stores it in an isolation area for transaction data that fails the integrity check or the legality check. When multiple consecutive exceptions are detected or the amount of abnormal data exceeds a preset threshold within a specified time, an alarm mechanism is triggered and the abnormal status is synchronized to the central server in real time for further analysis and processing.

9. The method according to claim 1, characterized in that: Also includes: S90: When the edge device is interrupted due to a fault, the fault recovery mechanism is enabled to reload the unfinished transaction tasks; Transaction response data that is not synchronized to the central server is resent through the fault recovery mechanism.

10. A high-concurrency transaction data processing system based on edge computing, characterized in that: The system comprises: Multiple user terminals; At least one edge device, wherein the edge device is configured with: A receiving module, used to simultaneously receive transaction data from multiple user terminals through a multi-threaded concurrent receiving queue; A verification storage module is used to perform integrity verification and legality verification on the transaction data, and store the verified transaction data in a local cache; A data sharding module is used to shard the verified transaction data to generate multiple data shards, and intelligently distribute the data shards to multiple processing modules inside the edge device; A parallel processing module is configured with independent computing units for parallel processing according to the allocated data slices to generate processing results; The result reduction module uses a real-time parallel reduction algorithm to intelligently integrate the processing results obtained by each processing module to generate transaction response data; A data transmission module, used to compress the transaction response data using an adaptive compression algorithm; synchronize the compressed transaction response data to a central server through a multi-encrypted secure transmission channel; and feed back the synchronization status to the user terminal in real time; A data cleaning module, after confirming that the transaction response data is successfully synchronized to the central server, safely deletes the verified transaction data stored in the local cache; Central server, configured with: A data receiving module, used to receive transaction response data synchronized by edge devices; A data verification module is used to perform a secondary verification on the received transaction response data; A data storage module is used to persistently store transaction response data that has passed the secondary verification; A network communication device is used to establish a secure communication link between the user terminal, the edge device and the central server.

Images