Data processing method, system, computer device, readable storage medium and program product

Abstract

The application relates to a data processing method, a system, computer equipment, a computer readable storage medium and a computer program product, relates to the technical field of data processing, and can improve the processing efficiency of data. The method comprises the following steps: encapsulating acquired data into a data object in a preset form; determining a business logic rule corresponding to the data based on the data object; determining a module calling instruction according to the business logic rule, and calling a target external module to process the data object based on the module calling instruction; acquiring a module utility parameter of the target external module when processing the data object, and updating the business logic rule based on the module utility parameter to optimize subsequent data processing based on the business logic rule.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, system, computer device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence and distributed computing technologies, intelligent nodes, as the core of edge computing, cognitive Internet of Things and other systems, need to process increasingly complex data in terms of volume and modality, which places extremely high demands on the efficiency of their real-time data processing.

[0003] In existing technologies, intelligent nodes typically employ a tightly coupled architecture. Their internal data processing links are pre-defined, with functional modules deeply bound to specific data formats. When a node receives data in a non-preset format, it must undergo complex and time-consuming data format conversion and protocol adaptation before being processed by the corresponding functional module. Furthermore, due to the lack of a unified interaction standard between different functional modules, cross-module collaborative processing requires data to be converted multiple times between different proprietary protocols, resulting in significant performance degradation.

[0004] Therefore, how to improve data processing efficiency is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] Therefore, it is necessary to provide a data processing method, system, computer equipment, computer-readable storage medium, and computer program product to address the aforementioned technical problems.

[0006] In a first aspect, this application provides a data processing method applied to a data processing system, comprising:

[0007] The acquired data is encapsulated into a data object in a preset format;

[0008] Based on the data object, determine the corresponding business logic rules for the data;

[0009] Based on the business logic rules, determine the module invocation instruction, and based on the module invocation instruction, invoke the target external module to process the data object;

[0010] Obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

[0011] In one embodiment, the step of invoking a target external module to process the data object based on the module invocation instruction includes:

[0012] Based on the module invocation instruction, determine the identifier of the target external module;

[0013] Based on the identifier of the target external module, the service address of the target external module is obtained through a service discovery mechanism;

[0014] According to the preset interface contract, the service address is used to initiate a call to the target external module to drive the target external module to process the data object.

[0015] In one embodiment, determining the business logic rules corresponding to the data based on the data object includes:

[0016] Parse the semantic description of the data type in the data object, as well as the extended parameters used to pass context information;

[0017] When the semantic description and extended parameters satisfy the preset conditional expression, multiple intermediate business logic rules are matched from the preset rule set;

[0018] The processing priorities corresponding to the intermediate business logic rules are compared, and the intermediate business logic rule with the highest processing priority is determined as the business logic rule.

[0019] In one embodiment, the step of obtaining the module utility parameters of the target external module when processing the data object includes:

[0020] Get snapshots and timestamps of data objects before and after the module call;

[0021] The data processing accuracy is determined based on snapshots of the data objects before and after the module call;

[0022] The data processing delay is determined based on the timestamps before and after the module call;

[0023] Based on the data processing accuracy and data processing latency, the module utility parameters for processing the data object are determined.

[0024] In one embodiment, updating the business logic rule based on the module utility parameter includes:

[0025] Based on the module utility parameters, adjust the processing priority of the business logic rules; and,

[0026] When the processing priority meets the preset processing conditions, the target external module corresponding to the business logic rule is preloaded.

[0027] In one embodiment, the method further includes:

[0028] Based on the context identifier of the data processing system, the global identity identifier of the data processing system is determined;

[0029] An operation log is established based on the global identity identifier to obtain data object snapshots and timestamps from the operation log.

[0030] Secondly, this application also provides a data processing system, including:

[0031] The content encapsulation unit is used to encapsulate the acquired data into a data object in a preset format;

[0032] A rule decision unit is used to determine the business logic rules corresponding to the data based on the data object;

[0033] An extended interface unit is used to determine a module call instruction based on the business logic rules, and to call a target external module to process the data object based on the module call instruction.

[0034] The operation tracing unit is used to obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0036] The acquired data is encapsulated into a data object in a preset format;

[0037] Based on the data object, determine the corresponding business logic rules for the data;

[0038] Based on the business logic rules, determine the module invocation instruction, and based on the module invocation instruction, invoke the target external module to process the data object;

[0039] Obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

[0040] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0041] The acquired data is encapsulated into a data object in a preset format;

[0042] Based on the data object, determine the corresponding business logic rules for the data;

[0043] Based on the business logic rules, determine the module invocation instruction, and based on the module invocation instruction, invoke the target external module to process the data object;

[0044] Obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

[0045] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0046] The acquired data is encapsulated into a data object in a preset format;

[0047] Based on the data object, determine the corresponding business logic rules for the data;

[0048] Based on the business logic rules, determine the module invocation instruction, and based on the module invocation instruction, invoke the target external module to process the data object;

[0049] Obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

[0050] The aforementioned data processing methods, systems, computer equipment, computer-readable storage media, and computer program products encapsulate the acquired data into data objects of a preset format; determine business logic rules corresponding to the data based on the data objects; determine module invocation instructions according to the business logic rules, and invoke target external modules to process the data objects based on the module invocation instructions; obtain module utility parameters of the target external modules when processing the data objects, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules. In this application, by first encapsulating the acquired data into data objects of a unified preset format, the standardization of data form is achieved, thereby decoupling the rigid coupling between upper-layer business logic and lower-layer heterogeneous data sources; on this basis, the corresponding external modules can be matched and invoked for processing according to the content of the data objects, realizing the modularization of tasks; furthermore, the actual operational performance of external modules during the processing process is captured, and based on this, a feedback loop is formed to dynamically and continuously update the original business logic rules, achieving the technical effect that the decision logic of the data processing system can adaptively adjust and continuously optimize according to the operating effect, thereby improving the data processing efficiency in complex and ever-changing environments. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating a data processing method in one embodiment;

[0053] Figure 2 This is a flowchart illustrating the data processing method in another embodiment;

[0054] Figure 3 This is a block diagram of the data processing system in one embodiment;

[0055] Figure 4 This is a block diagram of the data processing system in another embodiment;

[0056] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] The data processing method provided in this application can be applied to a data processing system. In an exemplary embodiment, the system macroscopically includes a data processing system (also referred to as an intelligent node in this application) serving as the execution carrier, and a third-party module ecosystem that can dynamically interact with it. The core design idea of ​​the data processing system is to achieve architectural decoupling and evolvability, enabling it to dynamically discover, load, and invoke one or more external functional modules from the third-party module ecosystem based on the input heterogeneous data to complete the corresponding data processing tasks. Optionally, the system can be deployed as a physical edge computing node in an industrial IoT scenario or implemented as a virtualized software service in a cloud-native environment, exhibiting broad applicability.

[0059] In one exemplary embodiment, such as Figure 1 As shown, a data processing method is provided, applied to a data processing system, including the following steps S102 to S108. Wherein:

[0060] Step S102: Encapsulate the acquired data into a data object in a preset format.

[0061] The pre-defined data object can be a structured data structure used to achieve multimodal data coexistence. This data object can contain two parts: a raw data payload for storing the unprocessed raw data, and metadata describing the content, type, and characteristics of the raw data. Through this encapsulation method that binds data to its description, the data processing system can achieve a deep understanding of the data content, which is the technical foundation for subsequent content-driven decision-making.

[0062] For example, a data processing system can acquire raw data to be processed. It should be noted that the source and mode of the raw data are diverse. In one possible implementation, the data can be physical world signals collected by sensors on an edge device (e.g., temperature sensors, cameras), such as binary video stream data or temperature readings. In another possible implementation, the data can also be purely logical information, such as a JSON-formatted financial transaction record passed through a message queue in a financial auditing scenario.

[0063] After obtaining the raw data, it can be encapsulated into a predefined data object. The core purpose of this step is to standardize the data, laying the foundation for subsequent unified, content-driven rule matching and processing. Optionally, the predefined format can be a key-value pair structure, where one key stores the raw data and one or more other keys store descriptive information about that data. In another optional implementation, the data object can also be encapsulated as an XML (Extensible Markup Language) object, using different tags to distinguish between the raw data and its metadata. In one embodiment, data encapsulation can be performed using a local AI library according to the predefined format. Through this encapsulation step, regardless of the modality of the input raw data, the system transforms it into an internally circulating, uniformly standardized information container, thereby decoupling the subsequent processing logic from the specific data format.

[0064] Step S104: Based on the data object, determine the business logic rules for the corresponding data.

[0065] The data processing system can perform real-time monitoring and in-depth analysis of the standardized data objects generated in the previous step to determine the business logic rules they should follow. Specifically, the system has a pre-defined set of rules containing multiple business logic rules. Each rule defines one or more triggering conditions. The system determines whether a data object meets the triggering conditions of one or more rules by parsing its content, such as its descriptive information or certain characteristics (like data size).

[0066] In some embodiments, the business logic rule can be a structured data object that encapsulates a complete decision unit of a data processing system, including but not limited to preset conditional expressions used to logically match the attributes of the input data object to determine whether the rule is activated; an execution definition, which at least specifies the identifier of the target external module to be called and the parameters to be passed to the module; and a scoring object dynamically associated with the business logic rule for decision-making and optimization, which at least includes the processing priority reflecting its historical performance.

[0067] In one exemplary embodiment, the trigger condition for a business logic rule can be that the description information of a data object is video and its data size exceeds 1MB. When a data object that meets this condition flows into the system, the business logic rule can be matched and determined as the rule corresponding to the current data object.

[0068] Step S106: Determine the module call instruction according to the business logic rules, and call the target external module to process the data object based on the module call instruction.

[0069] Module invocation instructions are the direct basis for driving external functional modules to execute tasks. For example, for the rule matched in the previous example, the defined action might be to call the video compression module, so the system can generate a module invocation instruction containing the target module identifier (such as VideoCompressorV2).

[0070] For example, after determining the business logic rules, the system can generate a clear and executable module call instruction based on the actions defined by those rules. Subsequently, the system can use this module call instruction to invoke one or more target external modules to process the data object. External modules are functional units independent of the core of the data processing system and can be dynamically accessed in a standardized manner. For instance, the system can obtain the actual network address and call parameters of the video compression module by querying the local service registry or sending a request to a service discovery center (such as Nacos) on the network, based on the module identifier in the instruction. Finally, it can initiate a remote call through a predefined interface specification (such as gRPC or RESTAPI) to pass the data object to the module for compression processing.

[0071] Step S108: Obtain the module utility parameters of the target external module when processing data objects, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

[0072] The module utility parameters are a set of technical indicators used to quantitatively evaluate the performance of a module. In one optional implementation, it can be a Boolean value indicating whether the task was successful or failed. In another optional implementation, it can be the time consumed from the start to the end of the task. Optionally, in other embodiments, the module utility parameters may also include other performance indicators, such as resource consumption rate (e.g., CPU or memory usage), execution success rate, or data processing gain (e.g., compression ratio), to support more comprehensive optimization decisions.

[0073] For example, after obtaining the module utility parameter, the system can update previously triggered business logic rules based on this parameter. This update can adjust an inherent attribute of the rule. For instance, each rule can maintain a success rate counter. When the obtained module utility parameter indicates success, the system increments the success count for that rule. Through this continuous feedback and update, the system can quantitatively evaluate the actual performance of each business logic rule and the external modules it calls. The results of this update will be used to optimize subsequent data processing. For example, when the system faces the need to select one rule from multiple satisfying conditions for execution in the future, it can prioritize the rule with the highest historical success rate, thereby enabling the decision logic of the entire data processing system to adaptively adjust based on real-world performance.

[0074] In this embodiment, by first encapsulating the acquired data into data objects of a unified preset format, the standardization of data form is achieved, thereby decoupling the rigid coupling between the upper-layer business logic and the lower-layer heterogeneous data source. On this basis, the corresponding external modules can be matched and called for processing according to the content of the data object, realizing the modularization of tasks. Furthermore, the actual operational performance of the external modules during the processing is captured, and based on this, a feedback loop is formed to dynamically and continuously update the original business logic rules. This achieves the technical effect that the decision logic of the data processing system can adaptively adjust and continuously optimize according to the running effect, thereby improving the data processing efficiency in complex and ever-changing environments.

[0075] In an exemplary embodiment, the step of invoking a target external module to process a data object based on a module invocation instruction includes:

[0076] Based on the module call instruction, determine the identifier of the target external module;

[0077] Based on the identifier of the target external module, the service address of the target external module is obtained through the service discovery mechanism;

[0078] According to the preset interface contract, the service address is used to initiate a call to the target external module to drive the target external module to process the data object.

[0079] The module invocation command can be a structured command containing the target function's logical identifier, generated internally by the data processing system after the business logic rules are determined. The service discovery mechanism can be a network registry center used to dynamically maintain the mapping relationship between module logical identifiers and their actual network service addresses. The interface contract refers to a set of predefined standardized interface specifications followed by external modules, defining the communication methods, parameters, and data formats implemented by the modules. For example, the data processing system can flexibly and extensibly obtain the service address of the target external module. The service discovery mechanism includes, but is not limited to, the following implementation methods: In a distributed or cloud-native environment, the system can obtain the module's service address in real time by querying a dynamic service registry center (e.g., Nacos, Consul). During system deployment or initialization, the system can read a local static configuration file (e.g., an XML, YAML, or JSON file) that pre-records the logical identifiers of each external module and their fixed service addresses. In a local area network environment, the system can also dynamically discover available services in the network through a predefined network broadcast protocol (e.g., DNS-SD). Regardless of the discovery method used, subsequent calls will strictly follow the predefined interface contract.

[0080] Specifically, the data processing system receives and parses the module call instruction, extracting the logical identifier of the target external module (e.g., VideoCompressorV2). Then, using this logical identifier as an index, the data processing system queries the service discovery mechanism (e.g., the Nacos registry) to dynamically obtain the service address (e.g., IP address and port) currently registered by the module in the network. Finally, strictly adhering to the specifications defined by the interface contract (ModuleContract), the data processing system uses the obtained service address to initiate a standardized API call (e.g., a gRPC request) to the target external module, passing the data object to be processed as a parameter, thereby driving it to execute specific processing tasks.

[0081] Optionally, after resolving the module identifier, the data processing system (as a cloud service instance) can utilize service discovery mechanisms in a cloud-native environment (such as Kubernetes Service) to query and obtain the cluster internal service address of the discount computing microservice. Subsequently, the system also follows the interface contract (which can be represented as an OpenAPI / Swagger specification) and initiates a gRPC or RESTAPI call to that service address to complete the processing of the data object.

[0082] In this embodiment, the above steps achieve triple decoupling between the core processing logic and external functional modules in terms of identity, location, and implementation. This results in the external functional modules being plug-and-play, dynamically replaceable, and independently evolving. When the system needs to expand with new functions, no intrusive code modifications are required, thus greatly improving the functional scalability and architectural flexibility of the entire data processing system.

[0083] In one exemplary embodiment, based on the data object, the business logic rules for the corresponding data are determined, including:

[0084] Parse the semantic description of the data type in the data object, as well as the extended parameters used to pass context information; when the semantic description and extended parameters meet the preset conditional expression, match multiple intermediate business logic rules from the preset rule set; compare the processing priorities of the intermediate business logic rules, and determine the intermediate business logic rule with the highest processing priority as the business logic rule.

[0085] Among them, the semantic description, as a list of content of data objects, can be a structured metadata field. In one embodiment, through internally defined information such as dataType (such as VIDEO / TEXT enumeration values) and featureDim (feature vector dimension), it reveals the inherent type and technical characteristics of the raw data to the data processing system, which is the technical foundation for realizing content-driven decision-making.

[0086] Extended parameters, as accompanying work orders for data objects, can be a flexible collection of key-value pairs used to carry dynamic context information related to the current processing task.

[0087] A pre-defined conditional expression can be a logical statement written in advance when defining a rule to activate that rule. This expression performs Boolean operations on the semantic description and extended parameters of the data object as input variables, and the result determines whether the rule is matched.

[0088] The pre-defined rule set serves as a knowledge base for decision-making in the data processing system. It is a dynamically expandable library that includes all available business logic rules within the system. When making decisions, the system can perform global searches and matches within this set.

[0089] Intermediate business logic rules can be a temporary candidate set of matching rules formed when a data object simultaneously satisfies the conditional expressions of multiple rules in a preset rule set during a specific decision event.

[0090] In one specific implementation, the data processing system first performs deep content parsing on standardized data objects to read their semantic descriptions and extended parameters. Then, the system uses this parsed information to match the preset conditional expressions of each rule in its internal set of preset rules.

[0091] In a physical edge node scenario, when a data object encapsulating a video stream is processed, its semantic description might be {dataType:VIDEO}, and its extended parameter might be {temperature:1500}. This data object may simultaneously satisfy the conditional expressions of two intermediate business logic rules: rule A (dataType == VIDEO) and rule B (temperature > 1000).

[0092] In a cloud-based microservices scenario, when a data object encapsulating a financial transaction is processed, its semantic description might be {dataType:Financial_Transaction}, and its extended parameters might be {userLevel:VIP,amount>10000}. This data object may simultaneously match two intermediate business logic rules: rule C (userLevel == VIP) and rule D (amount>10000).

[0093] After matching one or more intermediate business logic rules, the data processing system can compare the processing priority scores of these rules. For example, in a physical scenario, if rule B (high temperature alarm) has a higher priority than rule A (video compression), the system will determine rule B as the final business logic rule. In a virtual scenario, if rule D (large-amount risk control) has a higher priority than rule C (VIP discount), the system will determine rule D as the final business logic rule.

[0094] Through the steps described above, the data processing system achieves intelligent decision-making based on deep content understanding. The system can dynamically select the optimal processing path from multiple alternative rules based on the inherent semantics of the data and its real-time context, greatly improving the accuracy, flexibility, and intelligence of the decision-making process.

[0095] In an exemplary embodiment, the step of obtaining the module utility parameters of the target external module when processing data objects includes:

[0096] Obtain snapshots and timestamps of the data objects before and after the module call; determine the data processing accuracy based on the snapshots of the data objects before and after the module call; determine the data processing latency based on the timestamps before and after the module call; and determine the module utility parameters when processing data objects based on the data processing accuracy and the data processing latency.

[0097] Among them, a data object snapshot can be a high-fidelity full-state record of a data object made by the data processing system at key time points when calling external modules (especially before and after the call). It is used to solidify the complete appearance of the data before and after processing, providing irrefutable evidence for subsequent auditing and analysis.

[0098] A timestamp is a time stamp recorded by a data processing system to precisely identify the moment an event occurs. In a specific implementation of this application, it can be at the nanosecond level to support high-precision performance analysis of the processing.

[0099] Data processing accuracy is a core utility parameter used to quantify the correctness of tasks performed by external modules. It is not simply a success / failure marker, but rather a comprehensive evaluation of the task's execution quality based on snapshots of data objects before and after the call, using pre-defined validation logic.

[0100] Data processing latency is another core utility parameter used to quantify the speed at which external modules execute tasks. It is calculated by the difference between the start and end timestamps of the module call, directly reflecting the execution efficiency of the task.

[0101] Specifically, before the system sends a data object to an external module, it may record a pre-invocation snapshot and a start timestamp. After the external module completes processing and returns a result, the system may record a post-invocation snapshot and an end timestamp again. Subsequently, the system may calculate specific utility parameters based on these records.

[0102] In the physical edge node scenario, when invoking a video compression module, the pre-invocation snapshot recorded by the system may include video data with a size of 2MB. After the invocation is completed, the post-invocation snapshot records compressed video data with a size of 800KB. The data processing system compares these two snapshots, for example, verifying whether the data size after invocation is smaller than that before invocation and whether the video format is valid, to determine that the data processing accuracy is successful. At the same time, the system determines the data processing latency by calculating the difference between the two timestamps.

[0103] In the cloud microservice scenario, when invoking an API for enriching user information, the pre-invocation snapshot may be a JSON object that does not contain the userLevel field. After the invocation is completed, the post-invocation snapshot is a JSON object with the newly added userLevel:VIP field. The data processing system verifies whether the expected field and value are correctly added in the post-invocation snapshot to determine the data processing accuracy. The time consumed for the API request and response is used to calculate the data processing latency through the timestamp difference.

[0104] Finally, the data processing system may combine the calculated data processing accuracy and data processing latency to form the final module utility parameter for this task.

[0105] In this embodiment, a refined physical examination report is generated for each operation from the two core dimensions of processing quality (accuracy) and processing efficiency (latency). This provides a high-quality, high-precision decision basis for the subsequent step of updating rules based on the utility parameters, thereby making the adaptive optimization process of the entire system more accurate and effective.

[0106] In an exemplary implementation, updating business logic rules based on module utility parameters includes:

[0107] adjusting the processing priority of the business logic rule based on the module utility parameter; and preloading the target external module corresponding to the business logic rule when the processing priority meets a preset processing condition.

[0108] The preset processing condition may be a logic threshold for triggering linkage optimization. It takes the processing priority of the rule as the judgment basis. When the condition is met, it indicates that the system has established a high degree of trust in the rule, and more optimization resources can be invested for it.

[0109] Preloading is a proactive performance optimization technique. Instead of waiting for a rule to be triggered before loading its corresponding external module, it loads the code and resources of the external module into the memory of the data processing system in advance after the preset processing conditions are met, putting it in a standby state, thereby eliminating or significantly reducing the cold start latency during subsequent actual calls.

[0110] For example, after obtaining the data processing accuracy in the module utility parameters corresponding to a certain rule, the data processing system can perform an action to adjust the processing priority. This adjustment process can be implemented using the Exponentially Weighted Moving Average (EWMA) algorithm, as shown in formula (1). The system can perform the following calculations:

[0111] First, the system obtains the data processing accuracy of this task as the current accuracy. current It reads the old priority of the rule from the rule base. old ).

[0112] The system can then determine the weighting factor α used for this calculation. Optionally, the value of α is dynamic. For example, when the historical comprehensive accuracy of a rule meets a preset threshold (e.g., ≥90%), α is set to a higher value (e.g., 0.5) so that the priority can more sensitively reflect changes in current performance.

[0113] Priority new =α·Accuracy current +(1-α)·Priority old (1)

[0114] Calculated Priority new (New priority) can override Priority old This becomes the new processing priority under this rule. Whether in physical edge node scenarios or cloud microservice scenarios, data processing systems use this unified algorithm to update their decision-making logic.

[0115] Optionally, the data processing system can also be used as a reinforcement learning agent to achieve more advanced intelligent decision-making oriented towards long-term rewards. The specific process is as follows: the system defines the current state as the semantic description and context information contained in the extended parameters of the current data object. For example, a state could be {dataType:IMAGE, context:low-light}. Furthermore, the system defines the action space as a list of all registered external modules capable of processing this state. For example, for the above state, the action space could be [FastDenoiseModule, HighQualityDenoiseModule].

[0116] Furthermore, when the system selects an action (i.e., invokes a module) in a certain state, it can calculate a reward value based on the acquired module utility parameters. For example, a utility parameter combining high accuracy and low latency can calculate a high positive reward value. The system continuously updates its internal decision-making strategy through a continuous state-action-reward learning loop. Through this process, the data processing system can learn an optimal strategy—that is, in which state, which action (which module to invoke) will yield the greatest long-term cumulative reward. This learned optimal strategy is the updated business logic rule.

[0117] After each priority update, the data processing system checks whether the preset processing conditions are met. If the conditions are met, the system can trigger a preloading mechanism to load or preheat the target external module corresponding to the high-priority rule, putting it in a standby state.

[0118] In this embodiment, by adjusting priorities, the system's decision-making logic can learn and evolve, dynamically tending to select processing paths with better historical performance, thereby improving the intelligence level of decision-making. Secondly, the preloading mechanism enables the system to anticipate high-probability tasks in the future and prepare resources in advance, thus significantly reducing execution latency. The combination of these two capabilities ultimately achieves the technical effect of making the data processing system both smarter and faster during operation.

[0119] In one exemplary embodiment, the method further includes:

[0120] Based on the context identifier of the data processing system, determine the global identity identifier of the data processing system; based on the global identity identifier, establish an operation log to obtain data object snapshots and timestamps from the operation log.

[0121] Context identifiers can refer to one or more inherent attributes that characterize the current operating environment of the data processing system. Global identity identifiers can be unique identifiers generated globally by the data processing system after integrating context identifiers and other information.

[0122] Specifically, at the beginning of the data processing method, the data processing system can first perform an identity creation step to determine its global identity identifier.

[0123] In a physical edge node scenario: the data processing system can obtain the MAC address or IEEE 802.15.4 ID of its network card as its context identifier. Subsequently, the system can fuse this identifier with a nanosecond-level timestamp and possibly combine it with a preset namespace (such as org.factory.area1) to finally determine a hierarchical global identity identifier, such as org.factory.area1.device001.

[0124] In a cloud-based microservices scenario: the data processing system (as a containerized instance) can obtain its container ID assigned by the container engine or its process identifier assigned by the operating system as its context identifier. Subsequently, the system can also combine this identifier with information such as timestamps to ultimately determine a unique global identity for a service instance.

[0125] Once a global identity is determined, it will be used to record all subsequent operations. Specifically, when the data processing system performs an operation and needs to log it (for example, during the process of obtaining module utility parameters), the system can create an operation log based on this global identity. This means that each newly generated log record can be automatically bound to this global identity (for example, the log may contain a field with nodeId: "org.factory.area1.device001"). Therefore, when subsequent steps retrieve data object snapshots and timestamps from the operation log, the retrieved data can be verifiably attributed to this unique node identity.

[0126] In this embodiment, the identity is forcibly bound to every core action of the system (logging, snapshotting, and timestamps). This ensures that all raw data used for utility parameter calculation and rule optimization can be accurately and non-repudiatedly attributed to its unique executing entity, greatly enhancing the security, auditability, and reliability of the entire self-optimizing system in a distributed environment.

[0127] In one exemplary embodiment, this application provides a data processing system for processing an incoming data entry, such as... Figure 2 As shown, the specific flow of the data processing method disclosed in this invention is fully described, and the flow includes:

[0128] S201. Identity creation and initialization.

[0129] During the startup or initialization phase of the data processing system, the system first performs an identity creation step to establish its unique and trusted identity within the network. Specifically, the data processing system can obtain its own context identifier. This identifier is an inherent attribute that uniquely represents its current operating environment. Subsequently, the system can fuse this context identifier with a high-precision timestamp to generate a globally unique global identity. This global identity will serve as the attribution basis for all subsequent operations, ensuring that all system actions can be accurately traced.

[0130] S202, Data Access and Standardized Encapsulation.

[0131] Once the identity is created, the data processing system begins receiving raw data from external sources. To mask differences in data sources and modalities, the system first performs a standardized encapsulation step. This encapsulates the acquired raw data into a predefined data object. This data object contains the raw data itself, a semantic description of its data type, and extended parameters for passing additional contextual information. Through this step, all heterogeneous input data is transformed into a standardized information container that can be uniformly recognized and processed within the system.

[0132] S203, Dynamic rule-based decision-making based on data objects.

[0133] Next, the data processing system makes intelligent decisions based on the standardized data objects generated in the previous step. The system can deeply analyze the semantic description and extended parameters of the data object and use the parsed information to match it with its internally maintained set of preset rules. During this process, the conditional expressions of multiple rules may be satisfied simultaneously, and these rules together constitute a candidate set of intermediate business logic rules. Subsequently, the system can compare the processing priorities of these candidate rules and determine the rule with the highest priority score as the final business logic rule to be executed in this task.

[0134] S204, Dynamic invocation of external modules based on contracts.

[0135] After determining the final business logic rules, the data processing system can generate a module call instruction based on these rules. The system first parses the instruction to identify the logical identifier of the target external module. Then, using this identifier as an index, the system queries and obtains the currently available network service address for that module through a service discovery mechanism. Finally, strictly adhering to a pre-defined interface contract, the system uses the obtained service address to initiate a standardized call to the target external module, passing the data object as a parameter to drive it to complete the specific processing task.

[0136] S205, Full-link traceability and utility parameter quantification.

[0137] While invoking external modules, the data processing system initiates a full-link traceability and utility parameter quantification process. First, the system establishes a unique operation log entry based on its global identity. In this log, the system records a snapshot of the data object before the call and its start timestamp, and again records a snapshot of the data object after the module returns its result and its end timestamp. Subsequently, the system analyzes this log entry, determining data processing accuracy by comparing the snapshots before and after the call, and determining data processing latency by calculating the difference in timestamps. Finally, the system combines these two quantified metrics to form the module utility parameters for this call.

[0138] S206. Optimize business logic based on module utility parameters.

[0139] Finally, the data processing system uses the module utility parameters generated in the previous step to complete a self-optimizing closed loop. The system can use these utility parameters as input and, through built-in optimization algorithms (such as EWMA or reinforcement learning), update the processing priority of the business logic rule that triggered the current call. After the priority is updated, the system can also check if it meets a preset processing condition. If it does, the system can trigger the preloading of the external module corresponding to that rule, preparing for the next efficient call. Through this continuous feedback and optimization cycle, the logic of the entire data processing system continuously improves and evolves during operation.

[0140] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0141] Based on the same inventive concept, this application also provides a data processing system for implementing the data processing method described above. The solution provided by this system is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing system embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0142] In one exemplary embodiment, such as Figure 3 As shown, a data processing system is provided, including: a content encapsulation unit, a rule decision unit, an extension interface unit, and an operation traceability unit, wherein:

[0143] The content encapsulation unit is used to encapsulate the acquired data into a data object in a preset format;

[0144] A rule decision unit is used to determine the business logic rules corresponding to the data based on the data object;

[0145] An extended interface unit is used to determine a module call instruction based on the business logic rules, and to call a target external module to process the data object based on the module call instruction.

[0146] The operation tracing unit is used to obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

[0147] In one embodiment, the extended interface unit is further configured to determine the identifier of the target external module according to the module call instruction;

[0148] Based on the identifier of the target external module, the service address of the target external module is obtained through a service discovery mechanism;

[0149] According to the preset interface contract, the service address is used to initiate a call to the target external module to drive the target external module to process the data object.

[0150] In one embodiment, the rule decision unit is further configured to parse the semantic description of the data type in the data object, as well as the extended parameters for transmitting context information;

[0151] When the semantic description and extended parameters satisfy the preset conditional expression, multiple intermediate business logic rules are matched from the preset rule set;

[0152] The processing priorities corresponding to the intermediate business logic rules are compared, and the intermediate business logic rule with the highest processing priority is determined as the business logic rule.

[0153] In one embodiment, the operation tracing unit is further configured to obtain data object snapshots and timestamps before and after the module call;

[0154] The data processing accuracy is determined based on snapshots of the data objects before and after the module call;

[0155] The data processing delay is determined based on the timestamps before and after the module call;

[0156] Based on the data processing accuracy and data processing latency, the module utility parameters for processing the data object are determined.

[0157] In one embodiment, the operation tracing unit is further configured to adjust the processing priority of the business logic rule based on the module utility parameter; and, when the processing priority meets the preset processing conditions, preload the target external module corresponding to the business logic rule.

[0158] In one embodiment, the identity identification unit is further configured to determine the global identity identifier of the data processing system based on the context identifier of the data processing system; and to establish an operation log based on the global identity identifier, so as to obtain data object snapshots and timestamps from the operation log.

[0159] In one embodiment, in an industrial IoT environment where the data processing system of the present invention is deployed, a smart node such as Figure 4 As shown, it is deployed next to a production line. The node's task is to monitor the high-definition video stream from the production line cameras and intelligently process and record it under specific conditions. The specific steps include:

[0160] When a smart node starts up, its Unique Identifier Unit (UID) is activated, and identity creation is performed. The UID can obtain the node's context identifier (such as an IEEE 802.15.4 device ID) and fuse it with a nanosecond-level timestamp and WGS84 latitude and longitude encoding to generate a hierarchical global identity identifier, such as org.factory.area1.device001. This identifier serves as the traceable root for all subsequent operations.

[0161] During operation, when sensor data is generated on the production line, the system's data access logic can invoke the Content Encapsulation Unit (CTU) to encapsulate this heterogeneous data into a standardized data object. An example of the specific structure of this CTU data object is as follows:

[0162]

[0163] In the above structure, the payload field stores the core raw data, i.e., the binary video stream in this example. The schema field serves as a semantic description, and its dataType key-value pair explicitly identifies the payload data type as video. The ext field, as an extended parameter, carries additional contextual information related to this task in a flexible key-value pair format, i.e., the temperature sensor reading {temperature:1500} in this example. Through this standardized encapsulation of CTU, all input data is transformed into an information container that can be uniformly recognized and processed within the system.

[0164] The encapsulated CTU data object is sent to a Relation Definition Unit (RDU) for intelligent decision-making. The RDU can deeply parse the schema and ext fields of the CTU object and use their contents to match a set of preset rules maintained internally. Assume that a rule in the set has a preset conditional expression defined as:

[0165] `ctu.schema.dataType === VIDEO && tu.ext.modelParams.temperature > 1000`. This conditional expression is a logical statement consisting of two parts: `ctu.schema.dataType === VIDEO` determines whether the data type is video, and `ctu.ext.modelParams`...

[0166] If temperature>1000, it is used to determine whether the temperature value in its attached context exceeds the threshold of 1000. The rule can only be matched if both conditions are met simultaneously.

[0167] Since the current data object meets this condition, the rule is matched. If multiple intermediate business logic rules are matched simultaneously, RDU will compare their processing priorities and select the one with the highest score as the final business logic rule. Based on the action defined by this rule, RDU then generates a module call instruction.

[0168] The module call instructions generated by the RDU are sent to the Extension Interface Unit (EIU) for execution. The EIU first parses the instructions to determine the module identifier, then obtains its service address through a service discovery mechanism, and finally initiates the call strictly following a predefined interface contract (ModuleContract). This contract defines the standardized methods for all external module implementations.

[0169]

[0170] The ModuleContract interface contract requires external modules to implement four core methods: the init method for receiving and initializing configuration; the process method for handling data processing, with inputs and outputs being standardized lists of CTU data objects; the getStatus method for externally querying the module's health status; and the destroy method for safely releasing resources when the module is unloaded. This contract ensures that EIU can interact with any module conforming to this specification in a unified and predictable manner.

[0171] Throughout the entire process of the EIU's call execution, the Traceability Record Unit (TRU) can bypass the monitoring and perform end-to-end tracing. The TRU can establish an operation log based on a "global identity" generated from the UID. An example of the log's structure is as follows:

[0172]

[0173]

[0174] This log structure comprehensively records the key information of an operation. The `opHash` field is a hash signature of the operation, used to prevent tampering. The `actor` field identifies the executor of the operation, including the called module ID and its version, as well as the node ID that initiated the call (i.e., the global identity identifier). The `snapshot` field records a snapshot of the data object before and after the call, clearly showing the data changes (e.g., compression from 2MB to 800KB). The `timeline` field records the start and end timestamps of the operation with nanosecond precision. After the module completes processing, the system can obtain the snapshot and timestamps from this log, and then calculate the data processing accuracy and data processing latency, ultimately combining them to form the module utility parameters.

[0175] Finally, the calculated module utility parameters can be fed back to the Relational Definition Unit (RDU) to complete the self-optimization closed loop. The RDU can use these utility parameters as input to its built-in optimization algorithm (such as EWMA) to update the processing priority of the business logic rule that triggered the current call. After the priority update, the RDU can also check whether it meets preset processing conditions. If so, the RDU can generate an internal instruction to drive the EIU to preload the external module corresponding to the rule, preparing for the next efficient call.

[0176] To further illustrate the universality of the invention, this embodiment will describe how the data processing method of the invention is applied to a financial risk control software agent deployed in a cloud-native environment (e.g., a Kubernetes cluster). In this scenario, the data processing system is a purely virtualized software service instance.

[0177] When a new service instance starts in a cloud container, its Unique Identifier Unit (UID) retrieves the instance's context identifier within the computing environment, such as a container ID or process identifier (PID). Subsequently, the UID is combined with this identifier, a high-precision timestamp, and its logical network address within the cluster to generate a globally unique global identity identifier for audit traceability, such as prod.aml-ent.instance-7f8d9c4c7b. When a new international money transfer transaction enters the risk control system via a message queue (such as Kafka), the system's Content Encapsulation Unit (CTU) module encapsulates the transaction's JSON data into a standardized CTU data object. In this object, the payload field is the raw JSON string of the transaction, and the schema field describes its data type.

[0178] The `Financial_Transaction` field carries important business context information such as the user session ID and client IP address. The Relationship Definition Unit (RDU) parses this CTU data object in real time. Assuming its internal rule expression is "transaction amount greater than $10,000, and client IP address belongs to a high-risk region list," the RDU matches this rule because the CTU meets this condition. Based on the action defined by this rule, the RDU generates a multi-step investigation process, first generating a module call instruction requesting a call to the internal Customer Risk Assessment microservice. Upon receiving this instruction, the Extended Interface Unit (EIU) uses the service discovery mechanism of the cloud-native environment (such as Kubernetes Service) to resolve the logical identifier "Customer Risk Assessment" into the actual service address of the microservice within the cluster. Subsequently, the EIU strictly follows the preset interface contract defined by the OpenAPI / Swagger specification, initiating a gRPC call to this address, passing the CTU object to obtain the customer's internal risk level.

[0179] During the aforementioned call process, the Traceability Unit (TRU) fully records this cross-service distributed transaction. The generated logs are marked with the globally established identity identifier prod.aml-agent.instance-7f8d9c from previous steps and include CTU data snapshots before and after the call to the customer risk assessment service, as well as nanosecond-level API call timestamps. Based on this information, the TRU calculates the module utility parameters for this call (such as API response latency and the accuracy of the returned data).

[0180] The Relationship Definition Unit (RDU) receives the aforementioned utility parameters. Suppose that in subsequent investigation processes, the RDU needs to call an external watchlist API, and the TRU feedback indicates that this API call has very high latency, becoming a bottleneck in the entire risk control process. The RDU can then update its business logic rules based on this persistent inefficiency feedback: for example, dynamically adjusting the strategy, no longer making real-time synchronous calls for non-high-risk or low-value transactions, but instead switching to asynchronous, batch calls. This significantly optimizes the overall system's throughput and performance without sacrificing core risk control capabilities.

[0181] Each module in the aforementioned data processing system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0182] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores business logic rule data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a data processing method.

[0183] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0186] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0187] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0188] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0189] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0190] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0191] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a data processing system, the method includes: The acquired data is encapsulated into a data object in a preset format; Based on the data object, determine the corresponding business logic rules for the data; Based on the business logic rules, determine the module invocation instruction, and based on the module invocation instruction, invoke the target external module to process the data object; Obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

2. The method according to claim 1, characterized in that, The step of invoking a target external module to process the data object based on the module invocation instruction includes: Based on the module invocation instruction, determine the identifier of the target external module; Based on the identifier of the target external module, the service address of the target external module is obtained through a service discovery mechanism; According to the preset interface contract, the service address is used to initiate a call to the target external module to drive the target external module to process the data object.

3. The method according to claim 1, characterized in that, The step of determining the business logic rules corresponding to the data based on the data object includes: Parse the semantic description of the data type in the data object, as well as the extended parameters used to pass context information; When the semantic description and extended parameters satisfy the preset conditional expression, multiple intermediate business logic rules are matched from the preset rule set; The processing priorities corresponding to the intermediate business logic rules are compared, and the intermediate business logic rule with the highest processing priority is determined as the business logic rule.

4. The method according to claim 3, characterized in that, The step of obtaining the module utility parameters of the target external module when processing the data object includes: Get snapshots and timestamps of data objects before and after the module call; The data processing accuracy is determined based on snapshots of the data objects before and after the module call; The data processing delay is determined based on the timestamps before and after the module call; Based on the data processing accuracy and data processing latency, the module utility parameters for processing the data object are determined.

5. The method according to claim 4, characterized in that, The step of updating the business logic rules based on the module utility parameters includes: Based on the module utility parameters, adjust the processing priority of the business logic rules; and, When the processing priority meets the preset processing conditions, the target external module corresponding to the business logic rule is preloaded.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Based on the context identifier of the data processing system, the global identity identifier of the data processing system is determined; An operation log is established based on the global identity identifier to obtain data object snapshots and timestamps from the operation log.

7. A data processing system, characterized in that, The system includes: The content encapsulation unit is used to encapsulate the acquired data into a data object in a preset format; A rule decision unit is used to determine the business logic rules corresponding to the data based on the data object; An extended interface unit is used to determine a module call instruction based on the business logic rules, and to call a target external module to process the data object based on the module call instruction. The operation tracing unit is used to obtain the module utility parameters of the target external module when processing the data object, and update the business logic rules based on the module utility parameters to optimize subsequent data processing based on the business logic rules.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images