Data processing method and device, equipment and medium

Abstract

The invention discloses a data processing method and device, equipment and a medium, and the method comprises the steps: determining a service type corresponding to message data, and the message data is transmitted by a second system; a mapping relation corresponding to the service type is determined, a target mapping relation is obtained, different service types correspond to different mapping relations, and the mapping relation is the mapping relation between the field in the first system and the field in the second system under the corresponding service type; performing field mapping on the message data based on the target mapping relationship to obtain mapped data; and performing data processing operation corresponding to the service type based on the mapped data to obtain processed data. In this way, the expandability of cross-system data interaction can be improved, and the maintenance cost is reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device and medium. Background Technology

[0002] In cross-system data interaction, a hard-coded static architecture is usually adopted, and data is encapsulated and isolated through the DTO (Data Transfer Object) pattern. All field processing logic is fixed in the code, and field processing is deeply coupled with business logic. Simple field changes require full-link modification and release. As the system complexity increases, it becomes increasingly difficult to maintain and expand, and cannot support the transmission of rich and complex business objects.

[0003] Therefore, how to improve the scalability of cross-system data interaction and reduce maintenance costs is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data processing method, apparatus, device, and medium that can improve the scalability of cross-system data interaction and reduce maintenance costs. The specific solution is as follows:

[0005] In a first aspect, this application provides a data processing method applied to a first system, comprising:

[0006] Determine the service type corresponding to the message data, wherein the message data is sent by the second system;

[0007] Determine the mapping relationship corresponding to the business type to obtain the target mapping relationship. Different business types correspond to different mapping relationships. The mapping relationship is the mapping relationship between fields in the first system and fields in the second system under the corresponding business type.

[0008] Based on the target mapping relationship, the message data is mapped to fields to obtain mapped data;

[0009] Based on the mapped data, perform data processing operations corresponding to the business type to obtain processed data.

[0010] Optionally, based on the mapped data, data processing operations corresponding to the business type are performed to obtain processed data, including:

[0011] Based on the mapped data, determine the data source and data processing operations required for the business type;

[0012] Load the target data based on the data source and the data processing operation;

[0013] The data processing operation is performed on the target data and the mapped data to obtain the processed data.

[0014] Optionally, based on the mapped data, the data source and data processing operations required for the business type are determined, including:

[0015] The mapped data is converted into a tree structure to obtain a field tree, where each node in the field tree is a field;

[0016] Based on the field tree, the data source and data processing operations required for the business type are determined.

[0017] Optionally, determining the mapping relationship corresponding to the business type to obtain the target mapping relationship includes:

[0018] Based on the configuration data in the configuration center, determine the mapping relationship corresponding to the business type to obtain the target mapping relationship;

[0019] The configuration data includes the mapping relationship between fields in the first system and fields in the second system under each business type.

[0020] Optional, also includes:

[0021] The configuration data is obtained through the interactive interface of the configuration terminal;

[0022] The configuration data is stored in the main database and slave database of the configuration center, and updated to the cache cluster.

[0023] Optional, also includes:

[0024] Before storing the configuration data in the main database of the configuration center, the historical data of the configuration data is snapshotted to obtain a first snapshot;

[0025] After updating the configuration data to the cache cluster, a snapshot of the configuration data is performed to obtain a second snapshot;

[0026] Associate the first snapshot with the second snapshot.

[0027] Optionally, after performing data processing operations corresponding to the business type based on the mapped data to obtain processed data, the method further includes:

[0028] The processed data is stored in the configuration center;

[0029] According to the first preset rule, the required data is read from the processed data in the configuration center;

[0030] The required data is processed based on the second preset rule to obtain the returned data. The second preset rule includes one or more of the following: access control rules, de-identification rules, protocol adaptation rules, cluster verification rules, and business service rules.

[0031] The returned data is then sent back to the client for display.

[0032] Secondly, this application provides a data processing apparatus for use in a first system, comprising:

[0033] A type determination module is used to determine the service type corresponding to the message data, wherein the message data is sent by the second system;

[0034] The relationship determination module is used to determine the mapping relationship corresponding to the business type and obtain the target mapping relationship. Different business types correspond to different mapping relationships, and the mapping relationship is the mapping relationship between fields in the first system and fields in the second system under the corresponding business type.

[0035] The field mapping module is used to perform field mapping on the message data based on the target mapping relationship to obtain mapped data.

[0036] The data processing module is used to perform data processing operations corresponding to the business type based on the mapped data to obtain processed data.

[0037] Thirdly, this application provides an electronic device, including a memory and a processor, wherein:

[0038] The memory is used to store computer programs;

[0039] The processor is used to execute the computer program to implement the aforementioned data processing method.

[0040] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned data processing method.

[0041] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the aforementioned data processing method.

[0042] As can be seen from the above scheme, the present invention provides a data processing method applied to a first system, comprising: determining the service type corresponding to message data, wherein the message data is sent by a second system; determining the mapping relationship corresponding to the service type to obtain a target mapping relationship, wherein different service types correspond to different mapping relationships, and the mapping relationship is the mapping relationship between fields in the first system and fields in the second system under the corresponding service type; performing field mapping on the message data based on the target mapping relationship to obtain mapped data; and performing data processing operations corresponding to the service type based on the mapped data to obtain processed data.

[0043] As can be seen, the beneficial effects of this application are as follows: A mapping relationship between fields under different business types across systems is pre-established. After receiving message data sent by the peer system, the business type corresponding to the message data is first determined, and then the mapping relationship under that business type is determined. Field mapping is performed to obtain the mapped data, which in turn yields data adapted to this system. Corresponding data processing operations are then performed. In this way, field mapping and business-related data processing operations are decoupled. When expanding business types, the corresponding mapping relationship and data processing operations can be configured, facilitating expansion, improving the scalability of cross-system data interaction, and reducing maintenance costs.

[0044] Accordingly, the data processing apparatus, device, readable storage medium, and product provided in this application also have the aforementioned technical effects. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0046] Figure 1 A flowchart of a data processing method provided in an embodiment of this application;

[0047] Figure 2 A configuration center processing flowchart provided in this application embodiment;

[0048] Figure 3 A timing diagram for configuration center processing provided in an embodiment of this application;

[0049] Figure 4 A timing diagram for message data processing provided in an embodiment of this application;

[0050] Figure 5 A flowchart for generating a field tree is provided in an embodiment of this application;

[0051] Figure 6 A sequence diagram for generating a field tree is provided in an embodiment of this application;

[0052] Figure 7 A data processing architecture diagram provided for an embodiment of this application;

[0053] Figure 8 This is a schematic diagram of a data processing device structure provided in an embodiment of this application;

[0054] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] First, the terms used in this application will be explained:

[0057] CAC: Configuration-as-Code, usually refers to the concept that configuration is code.

[0058] API: Application Programming Interface.

[0059] Entity: An entity object, which typically corresponds to a database table entity.

[0060] Redis: A caching technology used for managing data caching.

[0061] Generic Type Reflection: Reflection type technology.

[0062] Rules Engine.

[0063] DTO: A design pattern used to transfer data between different layers of an application.

[0064] In multi-domain, cross-platform (i.e., cross-system) data interaction, the vast majority adopt a hard-coded static architecture, using the DTO pattern for data encapsulation and isolation, and employing Entity database mapping for persistent output. Data interaction occurs between various application service layers. Even simple field changes require coordination across all consumers, and it cannot provide differentiated data for different clients. This architecture is not only overly rigid in format but also suffers from maintenance difficulties and poor scalability. Simple DTO designs, through fixed-format, indiscriminate serialization of field information, result in rigid and non-universal data formats. Data parsing using the JSONTO OBJECT format is prone to various errors due to data contract constraints. Business object scenarios are implemented singly in the code, with completely static output, resulting in overly simplistic content. Field behaviors are determined at compile time and cannot be adjusted at runtime. Instantiating objects through database tables into Entity object classes requires re-instantiating object classes for business expansion, lacking abstraction conditions.

[0065] Current technical solutions suffer from the following drawbacks: all field processing logic is fixed in the code, and the deep coupling between field processing and business logic leads to a fixed product roadmap; simple field changes require end-to-end modification and deployment, lacking field-level observability, and as system complexity increases, they quickly become difficult to maintain and expand, becoming a major source of technical debt; the supported business objects are relatively limited, unable to support the rich and complex data transmission of business objects across platforms. Therefore, this application provides a data processing solution that can improve the scalability of cross-system data interaction and reduce maintenance costs.

[0066] See Figure 1 As shown in the figure, this application discloses a data processing method applied to a first system, including:

[0067] Step S11: Determine the service type corresponding to the message data, wherein the message data is sent by the second system.

[0068] The first system and the second system can be different business systems, including both hardware and software. For example, the first system could be an enterprise management system, and the second system could be an invoice cloud system. The enterprise management system could include core business functions such as finance, supply chain, and manufacturing. The invoice cloud system could manage invoices, including functions such as issuing invoices, receiving invoices, verifying old invoices, and archiving invoices. Message data refers to the interaction data between the first system and the second system. For example, the enterprise management system could receive invoice data sent by the invoice cloud system, i.e., message data.

[0069] In this embodiment, the message data carries the business type, which can be a document type, such as a tax invoice, accounts receivable invoice, or accounts payable invoice. The business type can be represented by a unique identifier.

[0070] Step S12: Determine the mapping relationship corresponding to the business type to obtain the target mapping relationship. Different business types correspond to different mapping relationships. The mapping relationship is the mapping relationship between the fields in the first system and the fields in the second system under the corresponding business type.

[0071] It should be noted that fields may be represented by different characters in different systems. For example, in the second system, field F2 corresponds to field A1 in the first system. In this embodiment, a mapping relationship can be constructed between the fields in the first system and the fields in the second system under each service type, so as to realize the field mapping of message data and convert it into data adapted to the receiving system.

[0072] In an optional implementation, determining the mapping relationship corresponding to the business type to obtain the target mapping relationship includes: determining the mapping relationship corresponding to the business type based on the configuration data of the configuration center to obtain the target mapping relationship; wherein, the configuration data includes the mapping relationship between fields in the first system and fields in the second system under each business type.

[0073] The configuration center stores configuration data including mapping relationships. In this embodiment, mapping relationships can be retrieved from the configuration center for caching, and the mapping relationship corresponding to the business type can be determined from it to obtain the target mapping relationship. Alternatively, the mapping relationship corresponding to the business type can be directly read from the configuration center based on the business type. This pre-configuration in the configuration center and retrieval of mapping relationships when needed improves data processing efficiency.

[0074] Furthermore, in an optional implementation, the method may further include: obtaining the configuration data based on the interactive interface of the configuration terminal; storing the configuration data in the main database and slave database of the configuration center, and updating it to the cache cluster.

[0075] Users can submit configuration data through an interactive interface, the configuration terminal retrieves the data, and then it is persistently stored. The reliability of the configuration data is ensured through a master database, slave databases, and a cache cluster. The cache cluster can be a Redis cache cluster.

[0076] Furthermore, in an optional implementation, the method may further include: before storing the configuration data in the main database of the configuration center, performing snapshot processing on the historical data of the configuration data to obtain a first snapshot; after updating the configuration data to the cache cluster, performing snapshot processing on the configuration data to obtain a second snapshot; and associating the first snapshot with the second snapshot.

[0077] In other words, the embodiments of this application can support configuration data updates, and snapshot processing is performed before and after the update, and the two snapshots are managed to ensure that historical traces are traceable.

[0078] In this embodiment, the configuration center serves as the system configuration data carrier. It relies on database persistence, synchronizes master and slave databases, and updates to the Redis cache cluster to generate a globally unique configuration ID. It also sets the creation timestamp and creator information. Meanwhile, it avoids duplicate configurations during the configuration process, ensures the existence of resource references, and performs preliminary business rule checks. It saves a comparison snapshot before and after modification to ensure that historical traces are traceable.

[0079] See Figure 2 As shown, Figure 2 This application provides a configuration center processing flowchart. Users submit configuration data, and the configuration center performs configuration verification, including uniqueness checks, reference verification, and business rule checks. If verification fails, a logical error is identified, and the process is terminated. If verification passes, a configuration ID (i.e., identifier) ​​and metadata are generated, a snapshot before modification is saved, persisted to the main database, and synchronized with the secondary database and updated in Redis cache. The platform (system) is notified that the update is complete, a success message is returned, and the configuration takes effect. The uniqueness check checks whether the key-value pairs of configuration items in the configuration data are globally unique. The reference verification checks whether the external resources referenced by the configuration items exist and are valid. Business rule checks may include preset rules such as numerical range and format regular expressions. Metadata is the configuration data corresponding to the configuration ID. In an optional implementation, a globally unique configuration ID can also be generated before the uniqueness check.

[0080] Further, see Figure 3 As shown, Figure 3 This application provides a sequence diagram for a configuration center process. 1. User / administrator creates / modifies configuration; 2. Configuration center console submits configuration data; 3. Configuration core service generates a globally unique ID and adds a timestamp / creator; 4. If it's a newly created configuration, the configuration key's uniqueness is checked in the master database, and the uniqueness result is returned to the configuration core service. If it's a modified configuration, the configuration core service retrieves the current configuration from the master database and generates a snapshot before modification; 5. The configuration core service verifies the existence of resource references and performs business rule checks; 6. The configuration core service saves / updates configuration data to the master database, and the master database returns the operation result; 7. Configuration data is synchronized to the slave database asynchronously; 8. The cache cluster is updated asynchronously; 9. A snapshot after modification is recorded and paired with the snapshot before modification; 10. A success message is returned to the configuration center console; 11. The result is displayed on the user / administrator terminal.

[0081] Step S13: Based on the target mapping relationship, perform field mapping on the message data to obtain the mapped data.

[0082] In this embodiment, target key fields can be extracted from message data. Based on the target mapping relationship, these target key fields are mapped to obtain mapped data. The mapped data includes the field names of the fields in the first system and the corresponding field values ​​extracted from the message data. For example, if the field name in the second system is F2 and the value is x1, in the first system, the corresponding field is A1, and the mapped value is field A1: x1. The mapped data is the data adapted to the first system and can be processed by the processing logic of the first system.

[0083] Step S14: Perform data processing operations corresponding to the business type based on the mapped data to obtain processed data.

[0084] In this embodiment of the application, different business types correspond to different data processing operations, and the corresponding data processing operations can be called to obtain the processed data.

[0085] In an optional implementation, data processing operations corresponding to the business type can be performed based on the mapped data to obtain processed data, including: determining the data source and data processing operations required for the business type based on the mapped data; loading target data based on the data source and the data processing operations; and performing the data processing operations on the target data and the mapped data to obtain processed data.

[0086] The data source can be understood as the specified data item that needs to be read from a preset database. The preset database may contain data from different business objects (invoices). Under the corresponding business type, the data source from which the corresponding field needs to be retrieved can be specified. Data processing operations can include association operations, summation operations, cleaning operations, transformation operations, etc. Association operations can be understood as data related to the connected business system that needs to be retrieved from the database; for example, if it is associated with other documents, data from those other documents needs to be retrieved. This application embodiment can determine the data to be loaded based on the data source and the data processing operations, then load the target data, perform data processing operations on the target data and the mapped data, and obtain the processed data.

[0087] In an optional implementation, determining the data source and data processing operations required for the business type based on the mapped data includes: converting the mapped data into a tree structure to obtain a field tree, wherein the nodes in the field tree are fields; and determining the data source and data processing operations required for the business type based on the field tree.

[0088] In other words, this embodiment of the application can convert mapped data into field data with fields as nodes. A tree structure facilitates rapid traversal, thereby accelerating data processing. Key symbols, including commas, parentheses, and field names, can be identified in the mapped data to generate a token (i.e., word) sequence. Then, a field tree is generated based on the token sequence, where each token represents the smallest unit with a specific semantic meaning. A root node can be created. For each field item, the corresponding node is recursively constructed, and the attributes of each node are set. If a left parenthesis is encountered, a branch node is created, and the field list is recursively parsed until a right parenthesis is encountered, ultimately forming a tree structure. Attributes can be descriptive information for the nodes, including field values ​​and field types.

[0089] In this embodiment, a field mapping platform can be used to determine the service type corresponding to the message data; determine the mapping relationship corresponding to the service type to obtain the target mapping relationship; perform field mapping on the message data based on the target mapping relationship to obtain mapped data; and perform data processing operations corresponding to the service type based on the mapped data to obtain processed data. The field mapping platform is a pre-deployed module in the system and can be deployed in the systems of both interacting parties. The dynamic data processing model of the field mapping platform includes methods such as data preloading, cleaning, transformation, and association. For example, {"A"~:"value1","B":"value2","C":[{"List":"listValue"}]}, where A, B, C, and List are all field lists, mapping them to the business fields in this system allows the values ​​value1...listValue to be mapped to the business fields in this system. This embodiment can parse the mapped data, i.e., field strings, which can be in JSON format. By parsing the input field strings, it converts them into a structured field tree. For example, it parses id, name, posts(title, content), it traverses each character of the string, identifies key symbols such as commas, parentheses (), field names, etc., and generates a token sequence, such as: Field("id"), X, Field("name"), X, Field("posts"), L_X, Field("title"), X, Field("content"), R_X. Through syntax analysis, X represents a comma, L_X represents a left parenthesis, R_X represents a right parenthesis, and so on. It parses the field list and field items, iterates through multiple FieldItems (i.e., field items) until it encounters a right parenthesis. If it is a left parenthesis, it creates a branch node, recursively parses the sub-field list, creates a root node as a container for the entire tree, and recursively constructs the corresponding FieldNode for each field item, sets the Parent (root node) attribute of the node, and sets the attributes of each node, finally forming a tree structure. The sequence may contain lists, and a field in a list may contain lists or objects. Each node has its own attributes, such as field type, whether it is a parent node, etc.

[0090] Furthermore, based on the field tree, we analyze which data sources are needed, what associations need to be made, and what processing is required. Taking invoicing in an enterprise management system as an example, it's necessary to parse the field information of the invoice application form transmitted from the Invoice Cloud for use as a Golden Tax invoice in the enterprise management system. This is done by building a structure tree based on the field information returned by the invoice application form from the Invoice Cloud in the enterprise management system. This structure tree reflects the header, body, sub-document bodies, upstream and downstream document internal codes, and related documents of the business object being the Golden Tax invoice. With this information, we can construct data using the structure supported by the enterprise management system, extracting all the necessary association information for subsequent use. Simultaneously, it removes meaningless fields from the Invoice Cloud that are not applicable to the enterprise management system and transforms the fields in the structure tree into field formats supported by the enterprise management system. It can load all necessary data at once, avoiding N+1 or nested query issues. For example, for two tax invoices, it can load all the necessary data at once. This data can include information about the business system being integrated with, specifying the organization, the business partner, the currency, and the corresponding basic information, etc., determined based on the data source and related operations. The loaded raw data is processed, filtering out logically deleted data using rule-based filtering conditions, such as filtering by organization, business partner, and currency. The data is converted to a format more suitable for API output, such as timestamps to strings, enumeration values ​​to readable text, and image IDs to complete URLs (Uniform Resource Locators). It supports deeper calculations on fields, such as addition, subtraction, multiplication, division, summation, and maximum value extraction. The master data and related data are concatenated and assembled according to the field tree requirements, processed, and the final processed data is returned. The master data is the mapped data corresponding to the JSON message, and the related data refers to data related to the integrated business system.

[0091] See Figure 4 , Figure 4This document provides a sequence diagram for message data processing in an embodiment of this application. The client of the second system sends request message data to the first system. The first system receives the message through an API router / controller. The first system's execution plan generator parses the field parameter in the message data, queries the field / related configuration from the configuration center, returns the configuration rules (i.e., the aforementioned mapping relationship), generates the optimal execution plan (SQL, call list), and returns the execution plan. In this embodiment, the execution plan generator, data acquisition engine, and data processing pipeline are all components of the field mapping platform. The execution plan generator performs field mapping and data processing operations based on the data source and data processing operations required by the mapped data, and generates the corresponding execution plan. The controller is also a component of the field mapping platform, executing the data loading plan (e.g., executing an SQL query), returning the original data set, and delivering the original data and execution plan to the data processing pipeline. The data processing pipeline performs data processing, including data cleaning (e.g., permission filtering), data transformation (e.g., timestamp to date), and data association (e.g., memory concatenation), returning the processed complete data. After field filtering, the final JSON response is returned to the client of the first system for display through the API router / controller.

[0092] In this embodiment, the data output for scenario factory and business object management is performed in the field mapping platform. This is used to manage the unified formatted output of business data such as multiple business objects (invoices) and self-made document scenarios. A dynamic mapper is created through reflection mechanism for encapsulation processing. The dynamic mapper can be regarded as a processing factory for individual business scenarios (business objects). It can produce independently, but the output data format is a unified standard, that is, the Map returned by the dynamic mapper.<String,Object> Load a generic response wrapper ApiResponse <t>In this context, the specific type of generic T is Map.<String,Object> That is, ApiResponse <Map<String, Object> >, Object, or field tree, ensures that regardless of the internal data, it maintains a consistent format externally. See also Figure 5 As shown, Figure 5 This application provides a flowchart for generating a field tree. The input is a complete entity object T (i.e., the target key field in the message data), and the input is a field selector (i.e., the target mapping relationship) to a dynamic mapper, i.e., a scene factory. The dynamic mapper analyzes object T based on reflection, traverses all attributes (i.e., fields), filters according to the selector, recursively processes nested objects to obtain a field tree, and outputs a dynamic response mapping Map. See also... Figure 6 As shown, Figure 6 This application provides a sequence diagram for generating a field tree. The controller inputs a complete object and a field selector to the dynamic mapper. Each attribute is processed through reflection to obtain the attribute name. If the field is in the selector, the field is selected, and the attribute value is obtained. If the value is a simple type, it is stored in the result Map. If the value is a nested type, the nested object is processed recursively, and the dynamic Map is output.

[0093] Furthermore, after performing data processing operations corresponding to the business type based on the mapped data to obtain processed data, the process may further include: storing the processed data in a configuration center; reading the required data from the processed data in the configuration center according to a first preset rule; processing the required data based on a second preset rule to obtain return data, wherein the second preset rule includes one or more of the following: access control rules, data anonymization rules, protocol adaptation rules, cluster verification rules, and business service rules; and returning the return data to the client for display.

[0094] In this embodiment, the client is the client in the first system, capable of displaying relevant data for viewing by relevant personnel. This embodiment can read the required data from the processed data in the configuration center through a rule engine and according to a first preset rule. The first preset rule can be a field rule, representing the fields to be retrieved. The access control rule can be understood as defining user or system access permissions to data through identity authentication and authorization mechanisms, ensuring that data is only used by legitimate entities in a compliant manner. Through the access control rule, the data can be returned to the client for display. The desensitization rule is used to convert sensitive data (such as ID card numbers, mobile phone numbers, and bank card numbers) into an unrecognizable form through replacement, encryption, masking, etc., while retaining data usability. The protocol adaptation rule solves data interaction problems caused by incompatibility between different systems due to communication protocols (such as HTTP) or data formats (such as JSON, XML) through protocol conversion, message format standardization, etc. The business service rule can be understood as specific processing rules in the first system. Cluster verification rules use mechanisms such as heartbeat detection, resource monitoring, and consistency checks to verify the status of nodes, resource usage, and data consistency in a distributed cluster, ensuring stable cluster operation. They can also be linked with business service rules to trigger business degradation logic in the event of a cluster failure.

[0095] See Figure 7 As shown, Figure 7 This document provides a data processing architecture diagram for an embodiment of this application. The system can deploy a field mapping platform, a configuration center, a rule engine, a field filter, a second preset rule processor, and a client. The second preset rule processor may include access control rules, data masking rules, protocol adaptation rules, cluster verification rules, and business service rules. The field filter can be used for further field filtering. Data interaction with other systems is achieved through an API gateway. Both the first and second systems may include an API gateway, or neither may, with a third party providing the API gateway platform. Clients in the first system request APIs, which are forwarded by the API gateway to the second system. The second system sends corresponding message data back to the first system. The API gateway then transmits the message data to a dynamic processing module, which includes a field mapping platform, a configuration center, and a rule engine. The field mapping platform performs field mapping and data processing operations corresponding to business types, obtaining processed data which is stored in the configuration center. The rule engine retrieves the required data from the configuration center according to rules, processes it through the field filter and the second preset rules, and returns it to the client for display. The configuration center is a platform that carries the final result of field metadata. The field mapping platform performs data preloading, cleaning, transformation, and association in the steps before the field metadata presents the final result. It can read the data first, then perform redundant filtering on the data, and transform it into data that the system can recognize. Finally, it can associate the relevant information corresponding to the data (such as name, code, enumeration, etc.). This application can dynamically respond to business changes and realize a general data processing process through the configuration center. It does not need to develop a lot of related logic processing on the spot after obtaining the data, which seriously leads to a long development cycle.

[0096] The field mapping platform distributes data sets to the configuration center through data preloading, cleaning, transformation, and association. The scenario factory generates corresponding business objects (such as invoices) and introduces them into the rule engine, establishing a connection channel with the configuration center. Data is then filtered through multi-engine rule processing, including access control, data masking rules, protocol adaptation, verification clusters, and business services. Data interaction is then achieved by sending requests through the API gateway. This architecture's dynamic, configurable, and automated nature addresses the three core pain points of traditional API metadata processing (long change cycles, numerous version conflicts, and high manpower investment), upgrading to real-time effectiveness, isolated and controllable operation and maintenance, and ultimately achieving an upgrade from a rigid architecture to a flexible business capability platform, becoming a core infrastructure for enterprise digital transformation. Metadata refers to the basic data based on the business system; API metadata processing refers to the aforementioned processing of target key fields in message data.

[0097] This provides a diversified configuration center that manages dynamic configuration of API fields. Its core technology involves designing a clear and flexible configuration structure (rules) to accurately match the corresponding API. It utilizes listening and push mechanisms to achieve dynamic hot updates and possesses governance capabilities such as versioning, canary releases, and environment isolation. Local caching and disaster recovery mechanisms ensure high performance and high availability, aiming to achieve a centralized yet dynamic control approach. The configuration center allows for free setting of fields and, based on actual business needs, performs complex calculations and logical combinations before loading the entire configuration scheme in real time for API data interaction. The field mapping platform parses complex field strings into a structured field tree to clarify the hierarchy and relationships between fields. Based on the field tree analysis, it identifies the required data source, relationships, and processing steps, generating an optimal data acquisition execution plan (such as using JOIN queries). This efficiently loads all raw data from the database or other microservices at once, building a scalable data processing pipeline. The raw data undergoes a series of processing steps, and based on the initial field tree structure, the processed master data and related data are precisely concatenated and finally serialized. This processed, flattened data is then reassembled into a correct, hierarchical object tree. This process is similar to mapping the query results of multiple database tables to nested DTO objects. The field mapping is implemented in the platform's memory using multi-level recursive parent-child nodes via INode. <t>The interface defines a unified contract, including a `Children` collection property, forming a self-referencing generic structure to achieve tree-like data modeling of arbitrary depth while avoiding explosive memory growth. A dynamic mapper has the ability to map all entities based on different business objects. Depending on the scenario, it uses generic type reflection technology to dynamically construct concrete type instances at runtime, implementing generic-based lazy binding of plugins. Furthermore, it utilizes generic type parameters... <t>Ensure strongly typed interactions between plugin hosts and consumers. Generics. <t>Typically, a system contains documents A, B, or dynamic documents C, etc., where A, B, and C represent individual business objects. Interactions between platforms are not limited to interacting only with documents A, B, or C, but are based on generics. <t>This refers to the general entity structure representing these business objects. During system runtime, specific instantiation is performed based on the business objects interacting between platforms. Therefore, the metadata differs, but the output method is consistent. This enables metadata management. Metadata refers to the basic data based on the business system, which is the fundamental unit for all business operations. Interactions with other platforms based on this data are all equivalent data exchanges.

[0098] Furthermore, this embodiment can perform field-level operation tracking, recording the processing process and time consumption of each field, providing traceability and observability. It features hierarchical data log management (DEBUG, INFO, WARN, ERROR), real-time log processing, and support for rapid problem localization. It also implements data anonymization, encrypted transmission and storage, and end-to-end data protection. Furthermore, it configures backup strategies, combining regular backups with real-time synchronization, and provides a rapid recovery mechanism with minute-level service recovery capabilities.

[0099] This configuration management center supports real-time adjustments to business rules, reducing hard coding and repetitive development work. It eliminates the need for downtime or redeployment, allowing for rapid adaptation to market changes and iterative business needs. Response time is reduced from days to minutes. Fine-grained field-level control allows for flexible adjustments to data display strategies based on different scenarios. Furthermore, it supports seamless adaptation across multiple environments, implementing configuration isolation between environments to ensure that testing and verification do not affect online stability. Adaptive data format support facilitates data exchange across various scenarios, automatically adapting to the data format requirements of different clients. It supports over 95% of field mappings and format conversions, such as through addition, subtraction, multiplication, division, summation, averaging, and maximum / minimum operations. The system supports functions such as maximum and minimum, parentheses, and also allows for conversion of timestamps to strings, enumeration values ​​to readable text, and image IDs to complete URLs. This largely eliminates incompatibility issues caused by differences in data formats between systems. Field-level changes are isolated, meaning adjustments to a single field do not affect the overall system. Configuration replaces hard coding, reducing system complexity. Standardized interface design reduces integration and maintenance costs. It supports multiple business object scenarios, dynamically responding to business changes. Standardized interfaces facilitate third-party integration and ecosystem expansion. The open architecture supports plug-in functionality expansion; new business requirements necessitating data interaction between multiple platforms can be directly implemented online without further standardized development. Through strategies such as a configuration management center, field mapping platform, and business object scenario factory, it achieves an upgrade from a rigid architecture to a flexible business capability platform, providing an efficient, reliable, and energy-saving governance architecture and general solution for real-time interaction between multiple domain platforms.

[0100] Further, see Figure 8 As shown, this application provides a data processing apparatus applied to a first system, comprising:

[0101] The type determination module 11 is used to determine the service type corresponding to the message data, wherein the message data is sent by the second system;

[0102] The relationship determination module 12 is used to determine the mapping relationship corresponding to the business type and obtain the target mapping relationship. Different business types correspond to different mapping relationships. The mapping relationship is the mapping relationship between the fields in the first system and the fields in the second system under the corresponding business type.

[0103] Field mapping module 13 is used to perform field mapping on the message data based on the target mapping relationship to obtain mapped data;

[0104] Data processing module 14 is used to perform data processing operations corresponding to the business type based on the mapped data to obtain processed data.

[0105] In an optional implementation, the data processing module 14 may include:

[0106] The determination submodule is used to determine the data source and data processing operations required for the business type based on the mapped data;

[0107] A loading submodule is used to load target data based on the data source and the data processing operation;

[0108] The processing submodule is used to perform the data processing operation on the target data and the mapped data to obtain the processed data.

[0109] In an optional implementation, the determination submodule can be specifically used to convert the mapped data into a tree structure to obtain a field tree, wherein the nodes in the field tree are fields; and to determine the data source and data processing operations required for the business type based on the field tree.

[0110] In an optional implementation, the relationship determination module 12 is specifically used to: determine the mapping relationship corresponding to the business type based on the configuration data of the configuration center, and obtain the target mapping relationship; wherein, the configuration data includes the mapping relationship between fields in the first system and fields in the second system under each business type.

[0111] Furthermore, the device also includes:

[0112] The data acquisition module is used to acquire the configuration data based on the interactive interface of the configuration terminal;

[0113] The persistence module is used to store the configuration data in the main database and slave database of the configuration center, and update it to the cache cluster.

[0114] In an optional implementation, the persistence module may also be used to: perform snapshot processing on historical data of the configuration data before storing the configuration data in the main database of the configuration center to obtain a first snapshot; perform snapshot processing on the configuration data after updating the configuration data to the cache cluster to obtain a second snapshot; and associate the first snapshot with the second snapshot.

[0115] In an optional embodiment, the device may further include:

[0116] A data storage module is used to store the processed data into a configuration center;

[0117] The data reading module is used to read the required data from the processed data from the configuration center according to a first preset rule;

[0118] The rule processing module is used to process the required data based on a second preset rule to obtain returned data, wherein the second preset rule includes one or more of the following: access control rules, de-identification rules, protocol adaptation rules, cluster verification rules, and business service rules.

[0119] The data display module is used to return the returned data to the client for display.

[0120] As can be seen, the embodiments of this application pre-establish the mapping relationship of fields under different business types across systems. After receiving the message data sent by the peer system, the business type corresponding to the message data is first determined, and then the mapping relationship under that business type is determined. Field mapping is performed to obtain the mapped data, so as to obtain the data adapted to this system, and then the corresponding data processing operation is performed. In this way, the field mapping and the data processing operation corresponding to the business are decoupled. When expanding the business type, the corresponding mapping relationship and data processing operation can be configured, which is convenient for expansion, improves the scalability of cross-system data interaction, and reduces maintenance costs.

[0121] See Figure 9 As shown in the figure, this application discloses an electronic device 20, including a processor 21 and a memory 22; wherein, the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, the data processing method disclosed in the foregoing embodiments.

[0122] For details regarding the specific process of the above data processing method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0123] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0124] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26; wherein, the power supply 23 is used to provide operating voltage for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0125] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method disclosed in the foregoing embodiments.

[0126] For details regarding the specific process of the above data processing method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0127] Furthermore, embodiments of this application also disclose a computer program product, including a computer program / instructions, which, when executed by a processor, implement the data processing method disclosed in the foregoing embodiments.

[0128] For details regarding the specific process of the above data processing method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0130] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0131] The above provides a detailed description of a data processing method, apparatus, device, and medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.< / t> < / t> < / t> < / t> < / t>

Claims

1. A data processing method, characterized by, The application is applied to a first system and comprises: determining a service type corresponding to message data, wherein the message data is sent by a second system; determining a mapping relationship corresponding to the service type to obtain a target mapping relationship, wherein different service types correspond to different mapping relationships, and the mapping relationship is a mapping relationship between fields in the first system and fields in the second system under a corresponding service type; performing field mapping on the message data based on the target mapping relationship to obtain mapped data; performing a data processing operation corresponding to the service type based on the mapped data to obtain processed data.

2. The data processing method according to claim 1, characterized in that, The data processing operation corresponding to the service type based on the mapped data to obtain processed data comprises: determining a data source and a data processing operation required by the service type based on the mapped data; loading target data based on the data source and the data processing operation; performing the data processing operation on the target data and the mapped data to obtain processed data.

3. The data processing method according to claim 2, characterized in that, Determining a data source and a data processing operation required by the service type based on the mapped data comprises: converting the mapped data into a tree structure to obtain a field tree, wherein a node in the field tree is a field; determining a data source and a data processing operation required by the service type based on the field tree.

4. The data processing method according to any one of claims 1 to 3, characterized in that, Determining a mapping relationship corresponding to the service type to obtain a target mapping relationship comprises: determining a mapping relationship corresponding to the service type based on configuration data of a configuration center to obtain a target mapping relationship; wherein the configuration data comprises a mapping relationship between fields in the first system and fields in the second system under each service type.

5. The data processing method according to claim 4, characterized in that, The application further comprises: obtaining the configuration data based on an interactive interface of a configuration terminal; storing the configuration data to a master database and a slave database of the configuration center and updating to a cache cluster.

6. The data processing method according to claim 5, characterized in that, The application further comprises: performing snapshot processing on historical data of the configuration data to obtain a first snapshot before storing the configuration data to the master database of the configuration center; performing snapshot processing on the configuration data to obtain a second snapshot after updating the configuration data to the cache cluster; associating the first snapshot with the second snapshot.

7. The data processing method according to claim 4, characterized in that, After performing a data processing operation corresponding to the service type based on the mapped data to obtain processed data, the application further comprises: storing the processed data into a configuration center; reading demand data in the processed data from the configuration center according to a first preset rule; processing the demand data based on a second preset rule to obtain return data, wherein the second preset rule comprises one or more of an authority control rule, a desensitization rule, a protocol adaptation rule, a cluster verification rule, and a business service rule; returning the return data to a client for display.

8. A data processing apparatus, characterized by, The application is applied to a first system and comprises: a type determination module configured to determine a service type corresponding to message data, wherein the message data is sent by a second system; A relationship determining module is configured to determine a mapping relationship corresponding to the service type, and obtain a target mapping relationship. Different service types correspond to different mapping relationships, and the mapping relationship is a mapping relationship between fields in the first system and fields in the second system under a corresponding service type. A field mapping module is configured to perform field mapping on the message data based on the target mapping relationship, and obtain mapped data. A data processing module is configured to perform a data processing operation corresponding to the service type based on the mapped data, and obtain processed data.

9. An electronic device, comprising: A device includes a memory and a processor, wherein: The memory is configured to save a computer program. The processor is configured to execute the computer program to implement the data processing method in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, A computer program is saved, and the computer program is executed by a processor to implement the data processing method in any one of claims 1 to 7.

Images