Data processing system and method

Abstract

The invention provides a data processing method. The data processing method comprises the following steps: dynamically adapting a protocol of an external system; collecting source data from an external system; performing primary processing on the source data, removing redundant data, and retaining target data; providing an empty data table, performing secondary processing on the target data, writing the target data into the empty data table, and generating a real data table; processing the data in the real data table for three times to generate a declarated data table, and declaring the declarated data table to an external system; receiving an audit receipt from an external system; performing finance and taxation compliance auditing on the declarable data table according to a receipt result; and notifying an external system to complete subsequent operation. According to the data processing method provided by the invention, the protocol of the external system is dynamically adapted, the external system is flexibly connected, and the cooperation efficiency is improved; in addition, uneven resource allocation of the execution units is avoided through an intelligent scheduling algorithm, and efficiency and stability are improved under high concurrency. The invention further provides a data processing system used for executing the data processing method.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a data processing system and method. Background Technology

[0002] Currently, the cross-border e-commerce industry is in a phase of rapid development, involving multiple heterogeneous domestic and international systems, such as e-commerce platforms like Amazon and Shopee, internal enterprise ERP and WMS systems, and external government and financial systems like logistics, customs single window, e-taxation, and bank foreign exchange settlement. These systems have inconsistent interface protocols and data standards, leading to prominent "data silos," difficulties in cross-system data interaction, and limited collaboration efficiency. At the same time, cross-border e-commerce business exhibits significant peak and trough characteristics, with order volumes potentially surging dozens of times during peak promotional periods. Traditional centralized architectures face performance bottlenecks such as system crashes and data loss when dealing with high concurrency scenarios. At the data level, enterprises need to process more than 200 different data formats daily, covering diverse structures such as EDI, XML, and JSON, further increasing the complexity of data integration and processing. Therefore, building a highly compatible, highly elastic, and highly available technical architecture has become crucial for driving the continued development of cross-border e-commerce. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide a data processing method that can flexibly and quickly connect to external systems and process data efficiently and stably.

[0004] Another object of the present invention is to provide a data processing system, wherein the data processing method operates based on the data processing system.

[0005] A data processing method includes: dynamically adapting to the protocol of an external system; collecting source data from the external system; performing a first processing on the source data: removing redundant data and retaining target data; providing an empty data table; performing a second processing on the target data: writing the target data into the empty data table to generate a real data table; performing a third processing on the data in the real data table: generating a data table that can be declared and submitting it to the external system; receiving an audit receipt from the external system; determining whether the audit is passed based on the receipt result, wherein the step of determining whether the audit is passed based on the receipt result includes: if the audit is passed, performing a financial and tax compliance audit on the completed data table based on the audit pass receipt result; determining whether the financial and tax compliance audit is qualified, wherein the step of determining whether the financial and tax compliance audit is qualified includes: if the financial and tax compliance audit is qualified, notifying the external system to complete subsequent operations.

[0006] The step of determining whether the review is approved based on the receipt result further includes: if the review is not approved, the reason for the review failure is displayed to the system user, and after the system user makes corrections, the step returns to provide an empty data table, and the target data is processed a second time: the target data is written into the empty data table to generate a real data table; the step of determining whether the financial and tax compliance review is qualified further includes: if the financial and tax compliance review is not qualified, the reason for the review failure is displayed to the system user, and after the system user makes corrections, the step returns to perform a financial and tax compliance review on the completed declaration data table based on the approval receipt result.

[0007] Optionally, the step of dynamically adapting the protocol of the external system further includes: constructing a protocol template standard library; receiving a protocol to be identified and extracting a feature profile of the protocol to be identified; calculating the similarity value between the protocol to be identified and the protocol templates in the protocol template standard library; determining whether the similarity value is greater than or equal to a threshold; if the similarity value is greater than or equal to the threshold, then the protocol to be identified is identified as a known protocol, and the protocol with the highest similarity value to the protocol to be identified in the protocol template standard library is retrieved and interfaced with the external system; if the similarity value is less than the threshold, then the protocol to be identified is identified as an unknown protocol, triggering an alarm and starting a protocol plugin hot loading mechanism, allowing maintenance personnel to upload a protocol parser JAR package to identify the unknown protocol.

[0008] Optionally, the step of constructing the protocol template standard library further includes: collecting a set of historical protocol templates; extracting the feature profile of each protocol template, the feature profile including a structural feature vector, a set of keywords, an authentication type, and an average packet size; extracting the structural feature vector from the protocol content; extracting the set of keywords from the protocol header description; extracting the authentication type from the protocol header description; obtaining the average packet size of the data volume of each communication of the protocol; and embedding the feature profiles of multiple protocol templates into the protocol library, the protocol library being the protocol template standard library.

[0009] Optionally, the step of processing the source data once: removing redundant data and retaining the target data further includes: using sharding technology to perform multi-dimensional sharding processing on the source data, decomposing the source data into tasks to be assigned at the optimal granularity; determining whether the number of tasks to be assigned is greater than a threshold; if the determination result is yes, then using an intelligent scheduling algorithm to intelligently assign data processing tasks; if the determination result is no, then directly assigning the data processing tasks; executing the data processing tasks, removing redundant data, and retaining the target data.

[0010] Optionally, the step of intelligently allocating data processing tasks using an intelligent scheduling algorithm further includes: calculating the execution cost of the tasks to be allocated based on load cost, urgency cost, and data locality cost; calculating the execution cost of all tasks to be allocated to generate a cost matrix; using a greedy algorithm based on the cost matrix to quickly obtain an initial task allocation scheme; performing a neighborhood search based on the initial task allocation scheme to find an optimized task allocation scheme with lower execution cost; selecting the task allocation scheme with the lowest total execution cost within a set time and issuing an allocation instruction; determining whether the current tasks to be allocated have been allocated; if allocated, continuing to allocate the next task to be allocated; if not allocated, returning to the step of using a greedy algorithm based on the cost matrix to quickly obtain an initial task allocation scheme.

[0011] Optionally, the step of calculating the execution cost of the task to be assigned based on load cost, urgency cost, and data locality cost further includes: detecting the task to be assigned and obtaining the resources required by the task to be assigned; detecting the remaining resources of each execution unit; calculating the load cost based on the resources required by the task to be assigned and the remaining resources of each execution unit; detecting and calculating the remaining time of the task to be assigned; obtaining the execution time of a single subtask in the task to be assigned and calculating the total allowed execution time of the task to be assigned; calculating the urgency cost based on the remaining time of the task to be assigned and the total allowed execution time of the task to be assigned; and detecting whether the data of the task to be assigned has been cached locally in a certain execution unit and calculating the data locality cost.

[0012] Optionally, the step of detecting whether the data of the task to be assigned has been cached locally in the execution unit, and calculating the data locality cost: then, according to the cost model: Cost_ij = k1 * (TaskSize_j / NodeCapacity_i) + k2 * ((Deadline_j - CurrentTime) / TimeLeft_j) + k3 * (1 -DataLocality_ij), the execution cost of the task to be assigned is obtained, where, j represents the task to be assigned. i represents the execution unit. Cost_ij represents the execution cost. TaskSize_j represents the resources required for the task to be assigned. NodeCapacity_i represents the remaining resources of the execution unit. (TaskSize_j / NodeCapacity_i) represents the load cost. Deadline_j represents the deadline for the task j to be assigned. CurrentTime represents the current time. (Deadline_j - CurrentTime) represents the remaining time for the task to be assigned. TimeLeft_j represents the total allowed execution time for the task to be assigned. ((Deadline_j - CurrentTime) / TimeLeft_j) represents the urgency cost. (1 - DataLocality_ij) represents the data locality cost. k1, k2, and k3 represent the weight parameters of each cost item, and k1 + k2 + k3 = 1; The step of processing the data in the real data table three times: generating a data table that can be declared and submitting it to the external system further includes: determining whether the number of tasks to be declared is greater than a threshold; if the determination result is yes, then using an intelligent scheduling algorithm to intelligently allocate the declaration tasks; if the determination result is no, then directly allocating the declaration tasks.

[0013] This invention provides a data processing system for executing the data processing method, interfacing with external systems, and automating multi-source data collection, reporting, processing, and collaboration. The data processing system includes a job management system, a data reporting system, a tax compliance system, and a data storage repository. The job management system is responsible for communication and protocol adaptation with the external systems, as well as coordinating internal task scheduling, data processing, data table submission, and receipt processing. The data reporting system is responsible for organizing the data in the data tables and converting it into a reportable data table. The tax compliance system is responsible for tax risk control and auditing of the data in the data tables. The data tables are stored in the data storage repository.

[0014] Optionally, the job management system includes a task management module, an executor management module, a protocol management module, a operation monitoring module, a log center, and a system management module. The task management module publishes execution tasks to the executor management module, which then executes data processing tasks. The protocol management module communicates and interfaces with external systems, ensuring protocol adaptation for subsequent data transmission. The operation monitoring module monitors the operational status of the executor management module. The log center records logs generated during the execution of tasks by the executor management module. The system management module manages other modules. The executor management module includes six execution units, and the task management module assigns data processing tasks to these six execution units, which are designated as the first execution unit, the second execution unit, the third execution unit, the fourth execution unit, the fifth execution unit, and the sixth execution unit.

[0015] This invention provides a data processing method that dynamically adapts to the protocols of external systems. This not only efficiently and accurately identifies known protocols but also quickly expands to identify unknown protocols, solving the problem of protocol fragmentation in system interfacing. This allows for flexible integration with external systems and improves collaborative efficiency. Furthermore, the data processing method uses an intelligent scheduling algorithm to efficiently allocate tasks to various execution units, avoiding uneven resource allocation and significantly improving efficiency and stability under high concurrency. This invention also provides a data processing system upon which the data processing method operates, enabling flexible and rapid integration with external systems and efficient and stable processing of data tasks. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention.

[0017] Figure 1 This is a schematic diagram of a cross-border e-commerce data processing system and an external system provided in an embodiment of the present invention.

[0018] Figure 2 for Figure 1 A detailed schematic diagram of the cross-border e-commerce data processing system and external systems described in the embodiment.

[0019] Figure 3 for Figure 2 A detailed schematic diagram of the task management module and executor management module in the job management system.

[0020] Figure 4 This is a flowchart of a data processing method provided in an embodiment of the present invention.

[0021] Figure 5 This is a flowchart of a method for dynamically adapting external system protocols according to an embodiment of the present invention.

[0022] Figure 6 for Figure 5 A further flowchart of the dynamic adaptation to external system protocols.

[0023] Figure 7 for Figure 4 A further flowchart of step 906 in the data processing method.

[0024] Figure 8 The flowchart illustrates a method for an intelligent scheduling algorithm provided in an embodiment of the present invention.

[0025] Figure 9 for Figure 8 A further flowchart of the intelligent scheduling algorithm.

[0026] Figure 10 for Figure 4 A further flowchart of step 910 of the data processing method, which involves submitting data to an external system. Detailed Implementation

[0027] To make the objectives, technical solutions, and effects of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention.

[0028] See Figure 1 and Figure 2This invention provides a data processing system 999, used to interface with an external system 888 in cross-border e-commerce operations, automating multi-source data collection, declaration processing, and collaboration. The data processing system 999 includes a job management system 100, a data declaration system 200, a financial and tax compliance system 300, and a data storage system 400. The external system 888 includes a source data system 810 and a third-party monitoring system 820. The job management system 100 is responsible for communication and protocol adaptation with the source data system 810 and the third-party monitoring system 820. The job management system 100 is also responsible for data table submission and receipt processing, as well as coordinating internal task scheduling and data processing. The job management system 100 includes a source data system interface 101 and a third-party monitoring system interface 102. The operation management system 100 retrieves source data from the source data system 810 via the source data system interface 101. The data in the source data system 810 originates from e-commerce platforms 811 (such as Amazon and Shopee), e-commerce ERP systems 812 (such as Wangdianguanjia and Jushuitan), WMS systems 813 (domestic and overseas warehouses), and logistics systems 814 (such as SF Express and 4PX). The operation management system 100 connects with the third-party supervision system 820 via the third-party supervision system interface 102, submitting data tables to the third-party supervision system 820 for review and receiving receipts from the third-party supervision system 820. The third-party supervision system 820 includes the General Administration of Customs Single Window System 821, the electronic tax system 822, and the local port administration system 823. The data declaration system 200 is responsible for organizing the data in the data tables and converting it into the officially required data declaration form format. The financial and tax compliance system 300 is responsible for the financial and tax risk control and review of the data in the data tables. The data repository 400 is a "data bus" connecting the operation management system 100, the data reporting system 200, and the financial and tax compliance system 300. Data tables are always stored in the data repository 400. In this embodiment, the source data originates from the platforms and systems listed above. In other embodiments, the source data of the source data system 810 may also originate from other platforms and systems, not limited to those listed in this embodiment.

[0029] See Figure 2The job management system 100 includes a task management module 110, an executor management module 120, a protocol management module 130, a runtime monitoring module 140, a log center 150, and a system management module 160. The task management module 110 publishes execution tasks to the executor management module 120. The executor management module 120 is the execution entity, responsible for source data acquisition and data processing. The protocol management module 130 records and retrieves the protocol content between the source data system 810 and the third-party monitoring system 820 and performs protocol matching to enable subsequent data transmission. The runtime monitoring module 140 monitors the running status of the executor management module 120, using Prometheus and Grafana to collect and visualize metrics such as CPU, memory, queue depth, and throughput of all executor units in real time, and sends alarms when anomalies occur. The log center 150 records the log content generated during the execution of the executor management module 120 and also provides distributed log aggregation, querying, performance analysis, and alarm functions. The system management module 160 manages other modules, provides a graphical interface, and allows for the setting and adjustment of system parameters for other modules.

[0030] The task management module 110 has an intelligent scheduling function. Based on the intelligent scheduling algorithm, it intelligently assigns data processing tasks to a specific execution unit of the executor management module 120. In the event of a surge in order volume, intelligent scheduling and task allocation ensure that each execution unit processes data in an orderly manner, avoiding problems such as system crashes and data loss.

[0031] See Figure 3 The executor management module 120 is the main task execution entity, composed of multiple elastically scalable containerized execution units. Each unit is an independent Docker container that can execute JAR packages and possesses complete data processing capabilities. Container orchestration is performed using Kubernetes, enabling dynamic horizontal scaling (auto scaling) based on queue load. In this embodiment, the executor management module 120 includes six execution units: the first execution unit 121, the second execution unit 122, the third execution unit 123, the fourth execution unit 124, the fifth execution unit 125, and the sixth execution unit 126. The six execution units are functionally equivalent, with no master-slave distinction. They all include functions such as data collection, data sharding, data processing, and reporting. Each execution unit can independently complete these tasks, receiving tasks assigned by the task management module 110 through a shared distributed message queue. The task management module 110 is the sole scheduler; the execution units do not communicate directly, avoiding complexity and achieving a high degree of decoupling and scalability. In other embodiments, the actuator management module 120 may also include other numbers of execution units, the number of which can be reasonably adjusted according to the capabilities of the hardware server and data processing requirements.

[0032] The protocol management module 130 can dynamically adapt to the protocol of the external system 888 to enable subsequent data transmission. Protocol adaptation between the protocol management module 130 and the external system 888 only needs to be performed once; once adapted, stable data transmission is possible. If the protocol of the external system 888 is updated, the protocol management module 130 automatically re-adapts the protocol. The source data system interface 101 and the third-party monitoring system interface 102 are located within the protocol management module 130. The protocol management module 130 uniformly handles communication with the source data system 810 and the third-party monitoring system 820, dynamically adapting the protocols of the source data system 810 and the third-party monitoring system 820, realizing protocol conversion and data standardization of heterogeneous systems. It also supports multiple protocol recognition and conversion, provides a hot-loading mechanism for protocol plugins, and allows new protocol integration without system downtime.

[0033] The data processing method of the present invention operates based on the aforementioned data processing system 999. Figure 4 The flowchart of the data processing method of the present invention includes the following steps 902-918.

[0034] Step 902: Dynamically adapt to the protocol of the external system. The protocol management module 130 dynamically adapts to the protocol of the external system 888 and completes protocol authentication to facilitate subsequent data transmission and interface with the external system 888.

[0035] Step 904: Collect source data from an external system. The job management system 100 collects source data from the source data system 810 through the source data system interface 101. The collected source data enters the task management module 110.

[0036] Step 906: Process the source data once: remove redundant data and retain the target data. The job management system 100 processes the source data once: including data cleaning, field mapping, format standardization, removing redundant data, retaining the target data, and writing the target data into the data repository 400.

[0037] Step 908: Provide an empty data table and perform secondary processing on the target data: write the target data into the empty data table to generate a real data table. The data storage repository 400 receives the target data and provides a predefined empty data table, the initial state of which is "empty". The data storage repository 400 performs secondary processing on the target data. Specifically, the secondary processing involves writing the target data into the empty data table to generate a real data table, at which point the table's state is updated to "initial filling state".

[0038] Step 910: The data in the real data table is processed three times: a data table ready for submission is generated and submitted to the external system. The data submission system 200 detects that the data table is in "initial filling state," retrieves the data from the real data table, and processes it three times: including business logic verification, generating a submission message, and filling the official template, generating a data table ready for submission that meets official requirements. At this point, the data table's status is updated to "ready for submission state." The job management system 100 detects that the data table is in "ready for submission state," and the six execution units 121-126 in the executor management module 120 connect to the third-party supervision system 820 through the third-party supervision system interface 102, submitting the data table in "ready for submission state" to the third-party supervision system 820 for review.

[0039] Step 912: Receive the audit receipt from the external system. The job management system 100 receives the audit feedback receipt from the third-party supervision system 820 in the external system 888 through the third-party supervision system interface 102.

[0040] Step 913: Determine whether the review is approved based on the receipt result. If the review is approved, proceed to step 914; if the review is not approved, proceed to step 915 and then return to step 908.

[0041] The third-party supervision system 820 sends an audit receipt to the data storage 400 through the third-party supervision system interface 102. The data storage 400 determines whether the audit is approved based on the receipt result. If the audit is approved, the status of the data table is updated to "completion of application" and the process proceeds to step 914.

[0042] Step 914: Based on the approved receipt, conduct a tax compliance review of the completed data table. The tax compliance system 300 performs a tax compliance review of the data table.

[0043] The data reporting system 200 detected that the data table was in the "reporting completed" state, triggering the tax compliance system 300 to conduct a tax compliance audit on the data table, such as export tax refund eligibility verification and fund compliance review.

[0044] Step 915: Display the reason for the failure to pass the review to the system user. After the system user makes the corrections, return to step 908.

[0045] Step 916: Determine if the financial and tax compliance audit is successful. If the financial and tax compliance audit is successful, proceed to step 917; if the financial and tax compliance audit is unsuccessful, proceed to step 918 and then return to step 914.

[0046] If the tax and financial compliance review is successful, the status of the completed data table will be changed to "Tax and Financial Compliance Status", and the process will proceed to step 917.

[0047] Step 917: Notify the external system to complete subsequent operations. After the financial and tax compliance review is approved, the operation management system 100 notifies the third-party supervision system 820 in the external system 888 of this final status of the data table to complete subsequent operations, driving it to complete the final operations, such as release, tax refund, etc.

[0048] Step 918: Display the reason for the audit failure to the system user. After the system user makes the corrections, return to step 914.

[0049] Step 902, namely, dynamically adapting to the protocol of the external system, is implemented through the protocol management module 130 of the job management system 100. Figure 5 The flowchart of the method for dynamically adapting to external system protocols of the present invention further includes steps 702-712.

[0050] Step 702: Build a standard library of protocol templates.

[0051] See Figure 6 Step 702, namely, constructing a standard library of protocol templates, further includes steps 721-727.

[0052] Step 721: Collect a set of historical protocol templates. Collect the protocol content and formats obtained from the external system 888 in the past, and compile them into a set of protocol templates.

[0053] Step 722: Extract the feature profiles for each protocol template. The feature profiles include four features: structural feature vector, keyword set, authentication type, and average packet size.

[0054] Step 723: Extract structural feature vectors from the protocol content. A structural feature vector is a sequence of numbers describing the protocol architecture; it represents the deep data structure of the protocol. Structural feature vectors have extremely high protocol discriminative power. In a specific embodiment, the structural feature vector is: [XML_tag_count, JSON_depth, field_separator_type]. Where XML_tag_count represents the average number of XML tags (e.g., ...) in the protocol data. <order> , <price>). The more tags, the larger the value. JSON_depth represents the nesting depth of JSON objects in the protocol data. The more complex the nesting, the larger the value. field_separator_type indicates what separator (such as comma ,, vertical bar |, or special character) is used to separate fields.

[0055] Step 724, extract the keyword set from the protocol header description. The keyword set is a set containing the most specific and common words of the protocol. Protocols can be quickly and preliminarily screened and identified through keywords. In a specific embodiment, the keyword set is: {"SOAPAction", "FUNCTION", "BEGIN", "tracking_number", "status"}. Among them, SOAPAction is the standard HTTP header field of the SOAP WebService protocol and is its decisive marker. FUNCTION may appear in database query protocols or some old interfaces. BEGIN is commonly found at the beginning of traditional messages such as EDI (Electronic Data Interchange).

[0056] Step 725, extract the authentication type from the protocol header description. The authentication type is the method used by the protocol for authentication. The protocol type range can be greatly narrowed down through the authentication type. In a specific embodiment, the authentication types are: BASIC, OAUTH2, WS-Security. Among them, BASIC is simple username and password authentication, usually appearing in the HTTP header in the form of Authorization: Basic <base64 encoded>. OAUTH2 is the token authentication commonly used in modern APIs, usually in the HTTP header in the form of Authorization: Bearer <token>It appears in a form. WS-Security is a more complex security authentication standard used for the SOAP protocol.

[0057] Step 726: Obtain the average packet size for each communication in the protocol. In a specific embodiment, the average packet size can be: 150 (bytes), 20480 (20KB), or 1048576 (1MB). A simple heartbeat detection or status query API might have a packet size of only a few hundred bytes. An order query API returning multiple data entries might have a packet size of tens of KB. An interface for batch uploading orders or synchronizing a product database might have packet sizes in the MB range. The data volume of interfaces for different business functions varies significantly, and this characteristic can very efficiently distinguish between small "query" packets and large "synchronization" packets.

[0058] Step 727: The feature profiles of various protocol templates are built into the protocol library, which is the protocol template standard library.

[0059] Step 704: Receive the protocol to be identified and extract its feature profile. The protocol management module 130 interfaces with the external system 888 to receive the protocol to be identified from the external system 888. The protocol management module 130 extracts the feature profile of the protocol to be identified in real time. The following is a sample code example of the protocol management module 130 extracting the feature profile of the protocol to be identified in a specific embodiment for reference: Java public ProtocolFingerprint extractFingerprint(byte[] data) { ProtocolFingerprint fp = new ProtocolFingerprint(); String content = new String(data, StandardCharsets.UTF_8); / / 1. Calculate the structural eigenvectors fp.structureVector = new double[]{ StringUtils.countMatches(content, "<"), / / XML feature values StringUtils.countMatches(content, "{"), / / JSON feature values StringUtils.countMatches(content, "\n") / / Line count feature }; StringUtils.countMatches(content, "<"): Counts the number of times the < appears in a string. This is very likely the opening symbol for an XML / HTML tag. The higher this value, the greater the likelihood that it is an XML-like protocol. StringUtils.countMatches(content, "{"): Counts the number of times the curly brace "{" appears. This is a marker for the beginning of a JSON object. The higher this value, the more likely it is to be a JSON object. StringUtils.countMatches(content, "\n"): Counts the number of newline characters, which can determine whether the protocol is "one-line" (such as some API responses) or "multi-line" (such as traditional EDI messages, CSV files); Using extremely simple and efficient calculations, the huge differences in the fundamental structure of different protocols were captured; / / 2. Extract keyword set (using a predefined keyword library for matching) fp.keywords = extractKeywords(content, predefinedKeywordLibrary); This line of code calls a method that compares the original content with a predefined keyword library; predefinedKeywordLibrary: This is a library containing hundreds of protocol-specific terms, for example: "SOAPAction" (SOAP protocol) "tracking_number" (logistics protocol) "order_id" (e-commerce protocol) "BEGIN:" (EDI protocol) The extractKeywords method scans the content and finds all the words that appear in this library, forming a set as features; Quickly capture "signature words" in the protocol content; / / 3. Identify Authentication Type fp.authType = detectAuthType(content); return fp; } This line of code calls another method that specifically analyzes authentication information from the data stream (especially the HTTP header); The detectAuthType method will look for the following patterns: Does it have Authorization: Basic...? → Determined as BASIC certification Does it have Authorization: Bearer...? → Determined as OAuth2 certification Does it have an API-Key: ... → Key authentication for a private API Does it contain complex XML Security tags? → Determine if it is WS-Security authentication. Authentication type is one of the most stable and standard components of a protocol; identifying the authentication type almost locks down a large class of protocols.

[0060] Step 706: Calculate the similarity value between the protocol to be identified and the protocol templates in the standard protocol template library. In this embodiment, the similarity value of the structural feature vector of the protocol to be identified is specifically calculated using the cosine similarity algorithm. The cosine similarity algorithm is very suitable for comparing numerical vectors. It determines the similarity in direction by calculating the cosine value of the angle between two vectors. For example, the vector of the unknown protocol is [105, 12, 5], and the vector of the JSON template in the library is [2, 98, 1]. Their numerical differences are large, so the cosine similarity will be very low. The similarity value of the keyword set of the protocol to be identified is calculated using the Jaccard similarity algorithm. The Jaccard similarity algorithm is very suitable for comparing sets. It calculates the ratio of the intersection to the union of two sets. For example, the keyword set of the unknown protocol is {"tracking_number", "status_text", "city"}, and the keyword set of the SF Express template is {"tracking_number", "status", "city"}. The intersection size is 2 (tracking_number, city), the union size is 4, and the Jaccard similarity = 2 / 4 = 0.5. The authentication similarity value takes the form of 0 or 1. Specifically, by identifying the authentication type of the protocol to be identified, if the authentication type of the protocol to be identified matches the authentication type of the protocol in the protocol standard template library, the value is 1; otherwise, the value is 0.

[0061] The weighted similarity matching algorithm is used to calculate the structural feature vector, keyword set, and authentication type to obtain the comprehensive similarity value of the protocol to be identified. The calculation formula is: similarity value = (W1 * structural similarity) + (W2 * keyword similarity) + (W3 * authentication similarity), where W1 + W2 + W3 = 1.

[0062] The weight W1 for structural similarity is 0.5 because the protocol structural feature vector is the most stable and least likely to change core feature.

[0063] The weight W3 for authentication similarity is 0.3 because the authentication type is also highly stable.

[0064] The weight W2 for keyword similarity is 0.2 because the keyword set is most likely to change or be customized.

[0065] By using a weighted system, the decision-making process of human experts is simulated, which places more emphasis on essential characteristics and less on variable characteristics, thus making the final decision more accurate.

[0066] Step 708: Determine whether the similarity value is greater than or equal to a threshold. If the determination result is yes, proceed to step 710; if the determination result is no, proceed to step 712. Match the similarity value with protocol templates in the protocol template standard library to determine whether the protocol to be identified is a known protocol. In this embodiment, the similarity threshold is 0.7, which is an empirical value. In other embodiments, the similarity threshold can also be other values.

[0067] Step 710: Determine that the protocol to be identified is a known protocol, and retrieve the protocol with the highest similarity value in the protocol template standard library to interface with the external system. If the similarity value is greater than or equal to 0.7, it is considered that there is a protocol template in the protocol template standard library that is the same as or similar to the protocol to be identified, and the protocol to be identified is determined to be a known protocol. Then, retrieve the protocol with the highest similarity value in the protocol template standard library to interface with the external system 888 to transmit data.

[0068] Step 712: The protocol to be identified is determined to be an unknown protocol, triggering an alarm and initiating a hot-loading mechanism for protocol plugins. This allows operations personnel to upload a protocol parser JAR file to identify the unknown protocol. If the similarity value is less than 0.7, the protocol to be identified is determined to be an unknown protocol, and it is added to the protocol template standard library for future use. Once a protocol is identified as unknown, an alarm is triggered and the hot-loading mechanism for protocol plugins is initiated. Operations personnel upload a new protocol parser JAR file, which the system dynamically loads to identify the unknown protocol, thus enabling interface with external systems. During this process, no service restart is required; the system immediately has the ability to process new protocols.

[0069] The following is a test report of the dynamic adaptation protocol to compare the advantages of the dynamic adaptation protocol of this invention compared with the traditional hard-coding method.

[0070] Test case: Simulate access to a new e-commerce platform whose API response format is MsgPack (a binary format) instead of the more common JSON.

[0071] Test results:

[0072] Conclusion: By constructing a standard library of protocol templates and using weighted similarity matching, the dynamic protocol adaptation technology can not only identify known protocols with high accuracy, but also infinitely expand the ability to identify unknown protocols through a hot-loading mechanism, fundamentally solving the problem of "protocol fragmentation" in system interfacing.

[0073] See Figure 7 Step 906, which involves processing the source data once: removing redundant data and retaining the target data, further includes the following steps 801-805.

[0074] Step 801: Use sharding technology to perform multi-dimensional sharding processing on the source data, decomposing the source data into tasks to be assigned at the optimal granularity.

[0075] Step 802: Determine if the number of tasks to be assigned is greater than a threshold. If it is greater than the threshold, proceed to step 803; otherwise, proceed to step 804. The task management module 130 determines whether the number of tasks to be assigned is greater than the threshold. The threshold is an empirical value; in this embodiment, it is set to 300. In other embodiments, the threshold can be adjusted based on the computing power of the execution unit, etc.

[0076] Step 803: Intelligent scheduling algorithm is used to intelligently allocate data processing tasks. Task management module 110 assigns data processing tasks to a specific execution unit of executor management module 120 according to the intelligent scheduling algorithm. Using the intelligent scheduling algorithm efficiently allocates tasks to computing resources, reducing latency, avoiding timeouts, increasing throughput, and preventing system crashes and data loss when order volume surges.

[0077] Step 804: Directly assign data processing tasks. In this embodiment, there are six execution units 121-126. When the number of tasks assigned to each execution unit is no more than 50, that is, when the total number of tasks to be processed by the six execution units 121-126 is no more than 300 (i.e., the above threshold is an empirical value of 300), data processing tasks can be directly assigned without enabling the intelligent scheduling algorithm.

[0078] Step 805: Perform data processing tasks to remove redundant data and retain the target data. Six execution units 121-126 perform data processing tasks to clean the data, remove redundant data, and write the target data into the data storage repository 400.

[0079] Step 803 involves intelligently allocating data processing tasks using an intelligent scheduling algorithm. The task management module 110 uses this algorithm to assign the data processing tasks to a specific execution unit of the executor management module 120. This step is implemented through the task management module 110 of the job management system 100. Figure 8 The flowchart of the intelligent scheduling algorithm of the present invention further includes the following steps 602-612.

[0080] Step 602: Calculate the execution cost of the task to be assigned based on load cost, urgency cost, and data locality cost.

[0081] See Figure 9 Step 602, which calculates the execution cost of the task to be assigned based on load cost, urgency cost, and data locality cost, further includes the following steps 502-516.

[0082] Step 502: Detect the tasks to be assigned and obtain the resources required for the tasks. The task management module 110 detects the task to be assigned j and obtains the TaskSize_j value corresponding to the task to be assigned j. The task to be assigned j is data from external systems, such as e-commerce platform orders, inventory, invoices, logistics system logistics orders, third-party supervision system customs declarations, manifests, etc. Each task to be assigned j includes several sub-tasks, such as 1000 e-commerce platform orders, with each order being one sub-task. The TaskSize_j value represents the resources required to execute task j, such as the number of CPU cores / GB of memory (e.g., 8U / 512M, 16U / 1024M).

[0083] Step 504: Detect the remaining resources of each execution unit. Detect the value of the remaining resource NodeCapacity_i for each execution unit i. Here, i is the execution unit number. In this embodiment, there are six execution units, and execution unit i represents any one of the first execution unit 121 to the sixth execution unit 126. NodeCapacity_i is the remaining resource of execution unit i. Each execution unit is the location of a physical server, and each execution unit can be placed in a different location, such as the United States, Hangzhou, Japan, etc.

[0084] Step 506: Calculate the load cost based on the resources required by the task to be assigned and the remaining resources of each execution unit. The load cost is (TaskSize_j / NodeCapacity_i), which is the ratio of resources required to execute task j to the remaining resources of execution unit i. The higher the load cost, the higher the load cost. By calculating the load cost, execution unit i is selected to execute task j, ensuring load balancing and preventing execution unit i from being overloaded.

[0085] Step 508: Detect and calculate the remaining time for the tasks to be assigned. Detect and calculate the remaining time for task j: Deadline_j - CurrentTime. Deadline_j represents the deadline for task j, and CurrentTime represents the current time. If task j is about to time out, the smaller the Deadline_j - CurrentTime value, indicating that task j is more urgent.

[0086] Step 510: Obtain the execution time of each individual subtask in the task to be assigned, and calculate the total allowed execution time of the task to be assigned. Obtain the execution time of each individual subtask in task j to be assigned, and calculate the total allowed execution time TimeLeft_j for task j to be assigned. The execution time of an individual subtask, such as the execution time of a single order or a single customs declaration, is recorded in the job management system 100 and can be directly obtained. The total allowed execution time TimeLeft_j for task j to be assigned is obtained by multiplying the execution time of an individual subtask by the number of subtasks in task j to be assigned.

[0087] Step 512: Calculate the urgency cost based on the remaining time of the tasks to be assigned and the total allowed execution time of the tasks to be assigned. The urgency cost is ((Deadline_j - CurrentTime) / TimeLeft_j), which is the remaining time of task j to be assigned divided by the total allowed execution time. The more urgent the deadline, the smaller this value is, thus prioritizing the scheduling of current tasks and ensuring that urgent tasks are processed first.

[0088] Step 514: Check whether the data of the task to be assigned is already cached locally in a certain execution unit, and calculate the data locality cost. Check whether the data of the task to be assigned j is cached locally in execution unit i, and obtain the value of DataLocality_ij, where DataLocality_ij takes the value of 0 or 1. If the data of the task to be assigned j comes from the United States, and the physical location of execution unit i is also in the United States, then the data of the task to be assigned j is already cached locally in execution unit i, and DataLocality_ij takes the value of 1; otherwise, it takes the value of 0. The data locality cost is (1 - DataLocality_ij), that is, if the data of the task to be assigned j is already cached locally in execution unit i, then DataLocality_ij takes the value of 1, 1-DataLocality_ij=0, indicating that the data locality cost is 0; otherwise, it is 1.

[0089] Step 516: Obtain the execution cost of the task to be assigned according to the cost model: Cost_ij = k1 * (TaskSize_j / NodeCapacity_i) + k2 * ((Deadline_j - CurrentTime) / TimeLeft_j) + k3 * (1 - DataLocality_ij). Calculate the execution cost of execution unit i executing the task to be assigned j according to the cost model. j represents the task to be assigned. i represents the execution unit. Cost_ij represents the execution cost. TaskSize_j represents the resources required for the tasks to be assigned. NodeCapacity_i represents the remaining resources of the execution unit. (TaskSize_j / NodeCapacity_i) represents the load cost. Deadline_j represents the deadline for task j to be assigned. CurrentTime represents the current time. (Deadline_j - CurrentTime) represents the remaining time for the task to be assigned. TimeLeft_j represents the total allowed execution time for the tasks to be assigned. ((Deadline_j - CurrentTime) / TimeLeft_j) represents the urgency cost. (1 - DataLocality_ij) represents the data locality cost. k1, k2, and k3 represent the weight parameters of each cost item, and k1 + k2 + k3 = 1.

[0090] In this cost model, the load cost weight k1 accounts for a large proportion, at 0.5; the urgency cost weight k2 accounts for a medium proportion, at 0.3; and the data locality cost weight k3 accounts for a small proportion, at 0.2.

[0091] Step 604: Calculate the execution cost of all the tasks to be assigned and generate a cost matrix. Periodically generate the cost matrix based on the calculated execution cost Cost_ij. Periodically (e.g., every second), calculate the cost of all tasks j to be assigned and the cost of executable unit i, i.e., periodically execute steps 502-516 to obtain the execution cost Cost_ij and form the matrix. This embodiment uses 1000 tasks and 6 execution units as an example; the generated cost matrix is ​​as follows:

[0092] Step 606: Based on the cost matrix, a greedy algorithm is used to quickly obtain an initial task allocation scheme. The task management module 110 traverses each task j to be allocated and uses a greedy algorithm to quickly obtain an initial allocation scheme, that is, simply assigns it to the execution unit i corresponding to the lowest currently calculated execution cost Cost_ij value.

[0093] Step 608: Perform a neighborhood search based on the initial task allocation scheme to find an optimized task allocation scheme with lower execution cost. Start multiple asynchronous threads to perform a neighborhood search based on the initial allocation scheme; that is, randomly swap the initial allocation execution units i of two tasks j, compare their execution cost values, and find an optimized allocation scheme with lower execution cost.

[0094] Step 610: Within a set time, select the task allocation scheme with the lowest total execution cost and issue an allocation instruction. Finally, within a set time, such as 50ms, select the allocation scheme with the lowest total execution cost from all found optimized allocation schemes, issue an allocation instruction, and finally allocate the task j to execution unit i.

[0095] Step 611: Determine whether all tasks to be assigned have been assigned. If all tasks have been assigned, proceed to step 612; otherwise, return to step 606.

[0096] Step 612: Continue to assign the next task to be assigned.

[0097] See Figure 10 Step 910 involves processing the data in the real data table three times: generating a data table that can be submitted and submitting it to the external system. The "submitting to the external system" further includes the following steps 806-808.

[0098] Step 806: Determine if the number of tasks to be submitted exceeds a threshold. If it does, proceed to step 807; otherwise, proceed to step 808. The task management module 130 determines whether the number of tasks to be submitted exceeds a threshold. The threshold is an empirical value; in this embodiment, it is set to 300. In other embodiments, the threshold can be adjusted based on the computing power of the execution unit, etc.

[0099] Step 807: Intelligent scheduling algorithm is used to intelligently allocate application tasks. Task management module 110 assigns application tasks to a specific execution unit of executor management module 120 according to the intelligent scheduling algorithm. The intelligent scheduling algorithm efficiently allocates tasks to computing resources, reducing latency, avoiding timeouts, increasing throughput, and preventing system crashes and data loss when order volume surges.

[0100] Step 808: Directly assign the application tasks. In this embodiment, there are six execution units 121-126. When the number of tasks assigned to each execution unit is no more than 50, that is, when the total number of tasks that the six execution units 121-126 need to process is no more than 300, i.e., the above threshold is the empirical value of 300, the application tasks can be directly assigned without enabling the intelligent scheduling algorithm.

[0101] Step 809: Execute the declaration task. The execution unit assigned to execute the declaration task submits the data table of eligible declarations to the third-party supervision system 820 in the external system 888 for review.

[0102] In particular, the intelligent scheduling algorithm in step 807, which intelligently allocates application tasks, and the intelligent scheduling algorithm in step 803, which intelligently allocates data processing tasks, use the same intelligent scheduling algorithm. Since the intelligent scheduling algorithm has been described in detail in step 803, it will not be repeated here.

[0103] The following is a test report on the intelligent scheduling algorithm to visually compare the improvement effect of the intelligent scheduling of this invention compared with the traditional round-robin scheduling.

[0104] Test case: When order volume surges instantaneously, 10,000 computational tasks (data cleaning, transformation, and reporting) are intelligently assigned to execution unit i.

[0105] Test results:

[0106] Conclusion: The intelligent scheduling algorithm dynamically balances load cost, urgency cost, and data locality cost through a cost model, avoiding uneven allocation of execution unit resources and significantly improving efficiency and stability under high concurrency.

[0107] This invention provides a data processing method that dynamically adapts to the protocols of external systems. This not only efficiently and accurately identifies known protocols but also quickly expands to identify unknown protocols, solving the problem of protocol fragmentation in system interfacing. This allows for flexible integration with external systems and improves collaborative efficiency. Furthermore, the data processing method uses an intelligent scheduling algorithm to efficiently allocate tasks to various execution units, avoiding uneven resource allocation and significantly improving efficiency and stability under high concurrency. This invention also provides a data processing system upon which the data processing method operates, enabling flexible and rapid integration with external systems and efficient and stable processing of data tasks.

[0108] Although descriptions and illustrations of relevant embodiments of the present invention have been given, those skilled in the art should understand that these descriptions and illustrations do not constitute a limitation on the scope of the present invention. Various changes in form and detail can be made to the present invention without departing from the concept and scope thereof. Therefore, the scope of this disclosure is not limited to the above embodiments, but should be determined by the claims and their equivalents.< / token> < / price> < / order>

Claims

1. A data processing method, characterized in that, The data processing method includes: Protocols that dynamically adapt to external systems; Collect source data from external systems; The source data is processed once: redundant data is removed, and the target data is retained; An empty data table is provided, and the target data is processed in a secondary manner: the target data is written into the empty data table to generate a real data table; The data in the real data table is processed three times: a declaration-ready data table is generated and declared to the external system; Receive audit receipts from the external system; The approval status will be determined based on the receipt result. The step of determining whether the review is approved based on the receipt result includes: if the review is approved, then conducting a financial and tax compliance review on the completed data table based on the approval receipt result; Determine whether the aforementioned financial and tax compliance audit is satisfactory; The step of determining whether the financial and tax compliance audit is qualified includes: if the financial and tax compliance audit is qualified, then notifying the external system to complete the subsequent operations.

2. The data processing method according to claim 1, characterized in that, The step of determining whether the review is approved based on the receipt result further includes: If the review fails, the system user is shown the reason for the failure. After the system user makes the correction, the system user is returned to the above steps to provide an empty data table. The target data is then processed again: the target data is written into the empty data table to generate a real data table. The step of determining whether the financial and tax compliance audit is qualified further includes: If the tax compliance audit fails, the system user will be shown the reason for the failure. After the user makes the necessary corrections, the system user will return to the previous step and conduct a tax compliance audit on the completed data table based on the approval receipt.

3. The data processing method according to claim 1, characterized in that, The step of dynamically adapting the protocol to the external system further includes: Build a standard library of protocol templates; Receive the protocol to be identified and extract the feature profile of the protocol to be identified; Calculate the similarity value between the protocol to be identified and the protocol templates in the protocol template standard library; Determine whether the similarity value is greater than or equal to the threshold; If the similarity value is greater than or equal to the threshold, the protocol to be identified is identified as a known protocol, and the protocol with the highest similarity value to the protocol to be identified in the protocol template standard library is retrieved and interfaced with the external system. If the similarity value is less than the threshold, the protocol to be identified is determined to be an unknown protocol, triggering an alarm and starting the protocol plugin hot loading mechanism, allowing maintenance personnel to upload the protocol parser JAR package to identify the unknown protocol.

4. The data processing method according to claim 3, characterized in that, The step of constructing the standard library of protocol templates further includes: Collect a collection of historical protocol templates; Extract the feature profile of each of the protocol templates, the feature profile including structural feature vector, keyword set, authentication type, and average packet size; Extract the structural feature vector from the protocol content; Extract the set of keywords from the protocol header description; Extract the authentication type from the protocol header description; The average packet size is obtained for each communication data volume of the protocol. The feature profiles of various protocol templates are built into a protocol library, which is the protocol template standard library.

5. The data processing method according to claim 1, characterized in that, The step of processing the source data once, removing redundant data and retaining the target data, further includes: The source data is processed by multi-dimensional sharding using sharding technology, and the source data is decomposed into tasks to be assigned at the optimal granularity. Determine whether the number of tasks to be assigned is greater than a threshold. If the determination result is yes, then use an intelligent scheduling algorithm to intelligently assign data processing tasks; if the determination result is no, then directly assign the data processing tasks. Perform the data processing task to remove redundant data and retain the target data.

6. The data processing method according to claim 5, characterized in that, The step of intelligently allocating data processing tasks using an intelligent scheduling algorithm further includes: The execution cost of the task to be assigned is calculated based on load cost, urgency cost, and data locality cost. Calculate the execution cost of all the tasks to be assigned and generate a cost matrix; Based on the cost matrix, a greedy algorithm is used to quickly obtain an initial task allocation scheme; Based on the initial task allocation scheme, a neighborhood search is performed to find an optimized task allocation scheme with lower execution cost. Within the set time frame, select the task allocation scheme with the lowest total execution cost and issue the allocation instruction; Determine whether the current tasks to be assigned have been assigned. If they have been assigned, continue to assign the next task to be assigned. If they have not been assigned, return to the previous step and use a greedy algorithm based on the cost matrix to quickly obtain an initial task assignment scheme.

7. A data processing method according to claim 6, characterized in that, The step of calculating the execution cost of the task to be assigned based on load cost, urgency cost, and data locality cost further includes: Detect the task to be assigned and obtain the resources required by the task to be assigned. Detect the remaining resources of each execution unit; The load cost is calculated based on the resources required by the task to be assigned and the remaining resources of each execution unit. Detect and calculate the remaining time for the task to be assigned; Obtain the execution time of a single subtask in the task to be assigned, and calculate the total allowed execution time of the task to be assigned; The urgency cost is calculated based on the remaining time of the tasks to be assigned and the total allowed execution time of the tasks to be assigned. Detect whether the data of the task to be assigned has been cached locally in one of the execution units, and calculate the data locality cost.

8. The data processing method according to claim 7, characterized in that, After detecting whether the data of the task to be assigned has been cached locally in the execution unit and calculating the data locality cost: The execution cost of the task to be assigned is obtained according to the cost model: Cost_ij = k1 * (TaskSize_j / NodeCapacity_i) + k2 *((Deadline_j - CurrentTime) / TimeLeft_j) + k3 * (1 - DataLocality_ij). in, j represents the task to be assigned. i represents the execution unit. Cost_ij represents the execution cost. TaskSize_j represents the resources required for the task to be assigned. NodeCapacity_i represents the remaining resources of the execution unit. (TaskSize_j / NodeCapacity_i) represents the load cost. Deadline_j represents the deadline for the task j to be assigned. CurrentTime represents the current time. (Deadline_j - CurrentTime) represents the remaining time for the task to be assigned. TimeLeft_j represents the total allowed execution time for the task to be assigned. ((Deadline_j - CurrentTime) / TimeLeft_j) represents the urgency cost. (1 - DataLocality_ij) represents the data locality cost. k1, k2, and k3 represent the weight parameters of each cost item, and k1 + k2 + k3 = 1; The step of processing the data in the real data table three times: generating a data table that can be submitted and submitting it to the external system further includes: If the number of tasks to be submitted exceeds a threshold, the task is intelligently allocated using an intelligent scheduling algorithm. If the threshold is not met, the task is directly allocated.

9. A data processing system for executing the data processing method according to any one of claims 1-8, characterized in that, The data processing system includes a job management system, a data reporting system, a financial and tax compliance system, and a data storage system. The job management system is responsible for communicating with the external system and adapting protocols, as well as coordinating internal task scheduling, data processing, data table submission, and receipt processing. The data reporting system is responsible for organizing the data in the data tables and converting it into a reportable data table. The financial and tax compliance system is responsible for the financial and tax risk control and review of the data in the data tables. The data tables are stored in the data storage system.

10. A data processing system according to claim 9, characterized in that, The job management system includes a task management module, an executor management module, a protocol management module, a operation monitoring module, a log center, and a system management module. The task management module is used to issue execution tasks to the executor management module, which is used to execute data processing tasks. The protocol management module is used to interface and communicate with the external system, and to perform protocol adaptation to enable subsequent data transmission. The operation monitoring module is used to monitor the operating status of the executor management module. The log center records the log content generated by the executor management module during task execution. The system management module is used to manage other modules. The executor management module includes six execution units, and the task management module assigns data processing tasks to the six execution units, which are designated as the first execution unit, the second execution unit, the third execution unit, the fourth execution unit, the fifth execution unit, and the sixth execution unit.

Images