Unified label-based data processing method and system based on multi-source data environment

Abstract

The application relates to a unified label data processing method and system based on a multi-source data environment. The method comprises the following steps: calling an internal data service interface, matching a standard field dictionary according to a received service request, and obtaining service configuration information; querying whether the service configuration information exists through a database; querying a three-party source data interface service, processing the queried data according to a pre-configured load balancing and fuse strategy through the three-party source data interface service; performing data cleaning and conversion according to the internal and three-party source data association relationship of the service configuration information to obtain target data; and saving the target data according to metadata. The method has the advantages of improving field mapping efficiency and semantic consistency, reducing manual intervention, improving development efficiency, reducing three-party interface calling cost through cache reuse, and enhancing system fault tolerance and business continuity.

Description

Technical Field

[0011] ,

[0010] ,

[0009] , , ,

[0001] The present invention relates to the field of Internet technology applications, and particularly to a data processing method and system for unified labels based on a multi-source data environment. Background Art

[0002] In financial credit operations, enterprises need to interface with third-party data sources such as credit reporting agencies, social security platforms, and e-commerce platforms to obtain data such as user credit scores, income certificates, and consumption behaviors. However, the interfaces of different data sources have the following problems:

[0003] 1. Field naming differences: The same semantic field has different names in different interfaces (such as "CreditScore" and "Credit Score"); [[ID=...]]...Existing technologies mostly rely on manual configuration of field mapping rules, which have defects such as low efficiency, high maintenance costs, and poor scalability. The present invention proposes a solution that integrates semantic recognition and automated governance.

[0008] Currently, in view of the problems in the related technologies that rely on manual configuration of field mapping rules, resulting in low efficiency, high maintenance costs, and poor scalability, and being unable to solve the problems of multi-source interface field naming differences, structural heterogeneity, service instability, and high-frequency call costs, no effective solution has been proposed. Summary of the Invention

[0009] The purpose of the present invention is to address the deficiencies in the prior art and provide a data processing method and system for unified labels based on a multi-source data environment, so as to solve the technical problems in the related technologies that rely on manual configuration of field mapping rules, resulting in low efficiency, high maintenance costs, poor scalability, and being unable to solve the problems of multi-source interface field naming differences, structural heterogeneity, service instability, and high-frequency call costs.

[0010] To achieve the above purpose, the technical solution adopted by the present invention is:

[0011] ​​​​​​​​​​This invention provides a unified tag data processing method based on a multi-source data environment, comprising: receiving a business request by calling an internal data service interface; matching the business request with a standard field dictionary to obtain service configuration information; querying the database to see if the service configuration information exists; if the service configuration information does not exist in the database, querying a third-party source data interface service, caching the queried data in the database through the third-party source data interface service, setting the cache time, and returning the data; wherein, caching the queried data in the database through the third-party source data interface service includes: processing the data through a pre-configured load balancing and circuit breaker strategy; cleaning and transforming the data according to the internal association between the service configuration information and the third-party source data to obtain target data; saving the target data according to metadata, and caching the target data.

[0012] Optionally, before receiving business requests by calling the internal data service interface, the method further includes: configuring unified label fields, wherein a standard field dictionary is configured to define standard label fields with unified semantics within the organization, the standard field dictionary including: field code, field name, field description, field specification, data type, and data example; configuring internal data services, which are the organization's internal data output specifications, including: data service code, standard output data structure, and instructions for using the internal API interface call service; wherein the data service code includes: various data service interface code values; the standard output data structure is a unified data structure for internal use of the data service output, and the field names in the unified data structure use unified label codes; configuring third-party source data services, wherein the third-party source data services are API interface services for connecting to external data sources, and the third-party source data... The service includes: third-party interface URLs and third-party output data structures; configuring the internal association with third-party source data, which links the internal data service to the third-party source data service, and establishing a mapping relationship between the internal standard output data structure and the third-party output data structure. Field mapping is established by first matching with the standard field dictionary; if the match score is greater than or equal to a threshold, a field mapping relationship is established; if the match score is lower than the threshold, the process proceeds to the rule engine configuration; if no mapping relationship has been established yet, it is established through manual matching. After the mapping relationship is established, the matched fields are sent to the intelligent tag recognition module for model training; and configuring service configuration parameters, which configure the data caching and load balancing of the internal data service. Data caching is used to configure the cache database and cache duration; load balancing is used to select round-robin, weighted, and master / slave configurations.

[0013] Optionally, service configuration information can be obtained by matching the business request with a standard field dictionary, including: extracting features from the business request based on natural language processing and a pre-trained semantic model to obtain the field names, comments, and sample values ​​of the data to be processed in the business request; and matching the field names, comments, and sample values ​​of the data to be processed with the standard field dictionary to obtain the service configuration information.

[0014] Optionally, after querying the database to see if the service configuration information exists, the method further includes: if the service configuration information exists in the database, then returning the corresponding cached data.

[0015] Optionally, the third-party source data interface service connects to third-party API interfaces to assemble and call data, call process, and data encryption / decryption.

[0016] Optionally, data processing using pre-configured load balancing and circuit breaker policies includes: when load balancing includes round-robin, weighted, and master-slave modes, in round-robin mode, business requests use the first data source for querying, and when a business request is initiated again, it uses the second data source for querying; in weighted mode, based on preset weights, when the number of calls to the first data source is less than a preset value, the first data source is used for querying; when the number of calls to the first data source is greater than a preset value, the second data source is used for querying; in master-slave mode, the first data source is set as the master data source, and the second data source is set as the slave data source. Business requests are executed through the first data source, and when the first data source service fails, the business request is sent to the second data source.

[0017] Furthermore, optionally, data processing through pre-configured load balancing and circuit breaker strategies includes: when the third-party source data service fails multiple times consecutively within a preset time, triggering interface circuit breaker, stopping requests to the third-party source data service, taking the third-party source data service offline, and issuing an abnormal warning for the third-party source data service until the third-party source data service recovers and the interface service of the third-party source data service is put back online.

[0018] Optionally, saving the target data based on metadata includes: saving the target data based on metadata, storing it in a metadata table, and obtaining the target data with a metadata structure, wherein the target data with a metadata structure includes: the data source of the target data, the business scenario of use, and the query time.

[0019] Furthermore, optionally, the metadata in the metadata table is used to record the business meaning, data lineage, and lifecycle status of fields; among which, the lifecycle status includes: active, archived, or obsolete; data lineage refers to the tracking and recording of the entire process of data from generation to final disposal.

[0020] This invention provides a unified tag data processing system based on a multi-source data environment, comprising: a data access module for receiving multi-source data and extracting field names, values, and structural information from the multi-source data; a unified tag dictionary module for storing a standard field dictionary, which includes: storing field codes, standard names, business semantics, data types, and sample values; an intelligent tag recognition module for matching the field names, values, and structural information of the multi-source data with the standard field dictionary based on natural language processing and a pre-trained semantic model to obtain matching results; a field standardization module for generating a mapping rule set for the matching results through a built-in rule engine and a manual verification interface; an interface collaboration module for automatically concatenating and formatting a multi-source field mapping table based on the mapping rule set to obtain calling instructions; a cache control module for executing calls based on the calling instructions to obtain calling results; a load balancing module for performing load balancing based on the calling results to obtain processing results; and a metadata governance module for recording the entire process of fields.

[0021] This invention employs the above technical solution, receiving business requests by calling an internal data service interface; matching the business requests using a standard field dictionary to obtain service configuration information; querying the database to check if the service configuration information exists; if the service configuration information does not exist in the database, querying a third-party source data interface service, caching the queried data in the database through the third-party source data interface service, setting the cache time, and returning the data; wherein, caching the queried data in the database through the third-party source data interface service includes: processing the data through pre-configured load balancing and circuit breaker strategies; performing data cleaning and transformation based on the internal association between the service configuration information and the third-party source data to obtain the target data; saving the target data according to metadata, and caching the target data. Compared with existing technologies, it has the following technical effects: improving field mapping efficiency and semantic consistency, reducing manual intervention, improving development efficiency, reducing the cost of third-party interface calls through cache reuse, and enhancing system fault tolerance and business continuity. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a unified label data processing method based on a multi-source data environment according to Embodiment 1 of the present invention.

[0023] Figure 2 This is a schematic diagram of a unified labeling data processing method based on a multi-source data environment according to Embodiment 1 of the present invention in the context of data governance.

[0024] Figure 3 This is a schematic diagram of a unified tag data processing system based on a multi-source data environment according to Embodiment 2 of the present invention;

[0025] Figure 4 This is a schematic diagram of the data service configuration of a unified tag data processing system based on a multi-source data environment according to Embodiment 2 of the present invention;

[0026] Figure 5 This is a schematic diagram of a unified label data processing system based on a multi-source data environment in data governance, according to Embodiment 2 of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0028] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0029] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0030] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units (elements) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or apparatus. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The terms “multiple” / “several” used in this application refer to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can indicate: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0031] Example 1

[0032] An illustrative embodiment of the present invention, such as Figure 1 As shown, Figure 1 This is a flowchart illustrating a unified tagging data processing method based on a multi-source data environment according to Embodiment 1 of the present invention. The unified tagging data processing method based on a multi-source data environment provided in this application includes:

[0033] Step S100: Receive business requests by calling the internal data service interface;

[0034] Optionally, before receiving the business request by calling the internal data service interface in step S100, the data processing method based on unified tags in a multi-source data environment provided in this application embodiment further includes: configuring unified tag fields, wherein, by configuring a standard field dictionary, standard tag fields with unified semantics within the organization are defined, and the standard field dictionary includes: field code, field name, field description, field specification, data type, and data example; configuring internal data services, which are the organization's internal data output specifications, and include: data service code, standard output data structure, and instructions for using internal API interface calls; wherein, the data service code includes: various data service interface code values; the standard output data structure is a unified data structure output by the internal data service, and the field names in the unified data structure use unified tag codes; configuring third-party source data services, wherein the third-party source data services are interfaces with external... The data source API interface service includes: third-party interface URLs and third-party output data structures; configuring the internal association with third-party source data, which links the internal data service to the third-party source data service, and establishing a mapping relationship between the internal standard output data structure and the third-party output data structure. Field mapping is established by first matching with the standard field dictionary. If the matching score is greater than or equal to the threshold, the field mapping is established; if the matching score is lower than the threshold, the rule engine configuration is entered. If no mapping relationship has been established yet, it is established through manual matching. After the mapping relationship is established, the matched fields are sent to the intelligent tag recognition module for model training; configuring service configuration parameters, which configure the data caching and load balancing of the internal data service. Data caching is used to configure the cache database and cache time; load balancing is used to select round-robin, weighted, and master / slave configurations.

[0035] Specifically, the data processing method based on unified tags in a multi-source data environment provided in this application embodiment can be applied to the financial credit field. The data processing method based on unified tags in a multi-source data environment provided in this application embodiment includes two parts:

[0036] Part 1: Data Service Configuration

[0037] Before executing step S100, data configuration needs to be performed on the backend management page. Data service configuration includes: internal unified tag dictionary, internal data service, third-party source data service, internal and third-party source data association relationships and service configuration; among these,

[0038] First, configure the unified label fields. The internal unified label dictionary is a standard label dictionary that defines the unified semantics within the company. It includes field codes, field names, field descriptions, field specifications, data types, and data examples. Define a unified internal label dictionary for each business field according to its semantics.

[0039] Configure internal data services. Internal data services are the internal data output specifications of the company (i.e., the organization in this application embodiment). They include data service codes (different data service interface code values), standard output data structures (unified data structures for internal use of data service output, with field names in the data structures using unified label encoding and semantic uniformity), and internal API interface call services, so that internal personnel can understand the business meaning of each field in the output.

[0040] Configure the third-party source data service. The third-party source data service is an API interface service that connects to external data sources. It includes the third-party interface URL, the third-party output data structure, and specifies which third-party source data interface to call.

[0041] Configure the relationship between internal and external data services. The relationship between internal and third-party source data is to associate internal data services with third-party source data services. Establish a mapping relationship between the internal standard output data structure and the third-party output data structure (for example, the field 'zs' from credit agency A is mapped to the internal field 'creditScore'). The field mapping relationship is first established by matching through the intelligent tag recognition module. If the matching score is greater than or equal to the threshold (e.g., ≥0.8), the field mapping relationship is established. If it is lower than the threshold, it enters the rule engine configuration. Finally, if no mapping relationship is established, manual matching is performed. After the mapping relationship is established, the matched fields are sent to the intelligent tag recognition module for model training to improve the success rate of the next matching.

[0042] Configure service parameters. Service configuration involves setting up internal data caching and load balancing mechanisms. The caching mechanism can be configured to use a caching database (such as Redis / MongoDB) and specify the cache duration. Load balancing mechanisms can be selected as round-robin, weighted, or master-slave.

[0043] Part Two: Data Governance

[0044] This application embodiment provides internal data governance services after the data service configuration is completed, that is, steps S100 to S110 are executed. In step S100, business requests are received by calling the internal data service interface.

[0045] Business departments call the internal data service interface, which is an API interface and standard data structure output provided to internal departments of the company to call third-party source data interface services.

[0046] Step S102: Based on the business request, match the standard field dictionary to obtain service configuration information;

[0047] Optionally, in step S102, the service configuration information obtained by matching the business request with a standard field dictionary includes: extracting features from the business request based on natural language processing and a pre-trained semantic model to obtain the field names, comments, and sample values ​​of the data to be processed in the business request; and matching the field names, comments, and sample values ​​of the data to be processed with the standard field dictionary to obtain the service configuration information.

[0048] Specifically, tags are obtained using a standard field dictionary with unified internal semantics, relevant service configuration information is queried based on internal data services, and then a third-party source data query is initiated based on the configuration information.

[0049] Step S104: Query the database to see if the service configuration information exists;

[0050] Optionally, after querying the database to see if the service configuration information exists in step S104, the data processing method based on a unified tag in a multi-source data environment provided in this application embodiment further includes: if the service configuration information exists in the database, then return the corresponding cached data.

[0051] Specifically, before querying third-party source data, a data caching function is used. This data caching function is an implementation of the cache configuration for the data configuration function. Based on the configuration, the corresponding cache library is used to store and query data. For example, using `getRedisData(String serviceCode, String request, long ttl)`, each time, it first checks if the data exists in the Redis database. If it exists, the cached data is output; otherwise, it queries the third-party source data interface service, caches the retrieved data in the Redis database with a set cache time, and then outputs the data. In other words, it queries the database to check if the service configuration information exists; if it exists, the corresponding cached data is returned; if it does not exist, step S106 is executed.

[0052] Step S106: If the service configuration information does not exist in the database, query the third-party source data interface service, cache the queried data in the database through the third-party source data interface service, set the cache time, and return the data; wherein, caching the queried data in the database through the third-party source data interface service includes: processing the data through pre-configured load balancing and circuit breaker strategies;

[0053] Optionally, the third-party source data interface service connects to third-party API interfaces to assemble and call data, call process, and data encryption / decryption.

[0054] Optionally, data processing using pre-configured load balancing and circuit breaker policies includes: when load balancing includes round-robin, weighted, and master-slave modes, in round-robin mode, business requests use the first data source for querying, and when a business request is initiated again, it uses the second data source for querying; in weighted mode, based on preset weights, when the number of calls to the first data source is less than a preset value, the first data source is used for querying; when the number of calls to the first data source is greater than a preset value, the second data source is used for querying; in master-slave mode, the first data source is set as the master data source, and the second data source is set as the slave data source. Business requests are executed through the first data source, and when the first data source service fails, the business request is sent to the second data source.

[0055] Furthermore, optionally, data processing through pre-configured load balancing and circuit breaker strategies includes: when the third-party source data service fails multiple times consecutively within a preset time, triggering interface circuit breaker, stopping requests to the third-party source data service, taking the third-party source data service offline, and issuing an abnormal warning for the third-party source data service until the third-party source data service recovers and the interface service of the third-party source data service is put back online.

[0056] Specifically, in this application embodiment, the third-party source data interface service is to interface with a third-party API interface to assemble and call data, call process, and data encryption and decryption.

[0057] During the process of initiating a call to the third-party source data interface service, load balancing and circuit breaking strategies were also implemented based on the service configuration in the data service configuration; among them,

[0058] Load balancing is a functional implementation used for data configuration. Different modes are used depending on the configuration strategy, including: round-robin, weighted, and master-slave.

[0059] Round-robin: This request uses data source A, and the next request uses data source B (that is, in the embodiment of this application, when the load balancing is in round-robin mode, the business request uses the first data source to query, and when the business request is initiated again, it uses the second data source to query).

[0060] Weights: A:3, B:7 (i.e., the preset weights in this application embodiment). Data source A is used when the number of times data source A is called accounts for less than or equal to 30% of the total number of calls (i.e., the preset value in this application embodiment), and data source B is used when the number of times data source A is called accounts for more than 30% of the total number of calls (i.e., in this application embodiment, according to the preset weights, when the number of times data source A is called accounts for less than the preset value, data source A is used for querying).

[0061] Primary / Backup: A is primary and B is backup. Normally, all requests are sent to data source A. When the service of A fails, the requests are automatically sent to data source B. (That is, in the load balancing mode of this application embodiment, when the load balancing is in primary / backup mode, the first data source is set as the primary data source and the second data source is set as the backup data source. The query of the business request is executed through the first data source. When the service of the first data source fails, the business request is sent to the second data source.)

[0062] Circuit breaking occurs when a third-party source data service experiences multiple consecutive anomalies within a certain period of time. In this case, the interface circuit breaker is triggered, requests to the third-party source data service are stopped, the third-party source data service is taken offline, and an anomaly warning is issued. The interface service is then put back online once the third-party source data service recovers. (That is, in this embodiment of the application, when a third-party source data service experiences multiple consecutive anomalies within a preset time, the interface circuit breaker is triggered, requests to the third-party source data service are stopped, the third-party source data service is taken offline, and an anomaly warning is issued until the third-party source data service recovers and the interface service of the third-party source data service is put back online.)

[0063] Step S108: Perform data cleaning and transformation based on the internal relationship between the service configuration information and the third-party source data to obtain the target data;

[0064] Specifically, the output data is obtained by querying third-party source data, and the data is cleaned and transformed according to the internal association between the data service configuration information and the third-party source data. The data cleaning and transformation is to transform and clean the data structure output by the third-party source data interface into a standard output data structure (i.e., the target data in this application embodiment) according to the internal association between the third-party source data and the third-party source data.

[0065] Step S110: Save the target data according to the metadata and cache the target data.

[0066] Optionally, saving the target data based on metadata in step S110 includes: saving the target data based on metadata, storing it in a metadata table, and obtaining the target data with a metadata structure, wherein the target data with a metadata structure includes: the data source of the target data, the business scenario of use, and the query time.

[0067] Furthermore, optionally, the metadata in the metadata table is used to record the business meaning, data lineage, and lifecycle status of fields; among which, the lifecycle status includes: active, archived, or obsolete; data lineage refers to the tracking and recording of the entire process of data from generation to final disposal.

[0068] Specifically, after converting to a standard output data structure, metadata is saved based on this data query. The standard source data metadata storage saves the data from each request, records the data source, the business scenario used, and the query time, and saves the metadata into a table to facilitate tracking the source of the data, the processing method, and data analysis.

[0069] After the metadata is saved, the standard output data is cached using the caching function, and then the standard output data is output to the business department.

[0070] In summary, the data processing method based on unified tags in a multi-source data environment provided in this application embodiment enables unified field identification across the entire platform, reducing communication costs and improving data governance and analysis capabilities. Intelligent tag recognition reduces the rate of manual error matching and reduces labor costs. Cache reuse improves system performance and reduces the cost of third-party calls. Load balancing and circuit breaking mechanisms enhance system fault tolerance and business continuity. Connecting to third-party source data interfaces is simpler and more convenient, reducing development workload.

[0071] like Figure 2 As shown, Figure 2 This is a schematic diagram of a unified label data processing method based on a multi-source data environment according to Embodiment 1 of the present invention, applied to data governance. The specific details of the unified label data processing method based on a multi-source data environment provided in this embodiment are as follows: A data query is received; the data configuration is queried; it is determined whether there is a cache. If so, the standard data result is returned directly. If not, the configuration for calling a third-party interface is obtained; one of the third-party interfaces A, B, and C is selected for use through load balancing and circuit breaking strategies. If the standard input parameter is converted to third-party A input parameter, then third-party A is called, and the output parameter of third-party A is converted to a standard output parameter. If the standard input parameter is converted to third-party B input parameter, then third-party B is called, and the output parameter of third-party B is converted to a standard output parameter. If the standard input parameter is converted to third-party C input parameter, then third-party C is called, and the output parameter of third-party C is converted to a standard output parameter. The data is then stored in the database after converting the output parameters of third-party A, B, or C to standard output parameters, so that the standard data result can be returned later through cached data processing.

[0072] The use cases of load balancing differ in the embodiments of this application. For example, when querying the location of GPS latitude and longitude, there are three data sources such as A, B, and C that can be used to query the location. The load balancer decides which data source to use to query the location.

[0073] This invention employs the above technical solution, receiving business requests by calling an internal data service interface; matching the business requests using a standard field dictionary to obtain service configuration information; querying the database to check if the service configuration information exists; if the service configuration information does not exist in the database, querying a third-party source data interface service, caching the queried data in the database through the third-party source data interface service, setting the cache time, and returning the data; wherein, caching the queried data in the database through the third-party source data interface service includes: processing the data through pre-configured load balancing and circuit breaker strategies; performing data cleaning and transformation based on the internal association between the service configuration information and the third-party source data to obtain the target data; saving the target data according to metadata, and caching the target data. Compared with existing technologies, it has the following technical effects: improving field mapping efficiency and semantic consistency, reducing manual intervention, improving development efficiency, reducing the cost of third-party interface calls through cache reuse, and enhancing system fault tolerance and business continuity.

[0074] Example 2

[0075] An illustrative embodiment of the present invention, such as Figure 3 As shown, Figure 3 This is a schematic diagram of a unified tag data processing system based on a multi-source data environment according to Embodiment 2 of the present invention, applied to the unified tag data processing method based on a multi-source data environment in Embodiment 1. The unified tag data processing system based on a multi-source data environment provided in this application includes:

[0076] The data access module 31 receives multi-source data and extracts the field names, values, and structural information of the multi-source data; the unified tag dictionary module 32 stores a standard field dictionary, which includes: stored field codes, standard names, business semantics, data types, and sample values; the intelligent tag recognition module 33 matches the field names, values, and structural information of the multi-source data with the standard field dictionary based on natural language processing and a pre-trained semantic model to obtain matching results; the field standardization module 34 generates a mapping rule set for the matching results through a built-in rule engine and a manual verification interface; the interface collaboration module 35 dynamically generates a multi-source field mapping table, automatically concatenates and converts the format according to the mapping rule set to obtain calling instructions; the cache control module 36 executes the call according to the calling instructions to obtain the calling results; the load balancing module 37 performs load balancing based on the calling results to obtain the processing results; and the metadata governance module 38 records the entire process of the fields.

[0077] Specifically, the unified tag data processing system based on a multi-source data environment provided in this application, corresponding to Embodiment 1, also includes two parts in application: the first part, data service configuration, and the second part, data governance. For example... Figure 4 and Figure 5 As shown, Figure 4 This is a schematic diagram of the data service configuration of a unified tag data processing system based on a multi-source data environment according to Embodiment 2 of the present invention; Figure 5 This is a schematic diagram of a unified label data processing system based on a multi-source data environment in data governance, according to Embodiment 2 of the present invention.

[0078] In this embodiment of the application, the data access module 31 supports access via multiple protocols such as HTTP / API / message queues and parses heterogeneous data formats such as JSON / XML.

[0079] Unified Tag Dictionary Module 32: Defines an internal standard field dictionary based on business requirements. The dictionary includes field codes, field names, field descriptions, field specifications, data types, and data examples.

[0080] Intelligent label recognition module 33: Based on natural language processing (NLP) and pre-trained semantic models, it extracts features from the original field names, annotations, and sample values, and matches them with the internal standard field dictionary;

[0081] Field standardization module 34: Built-in rule engine (regular expression, decision tree) and manual verification interface, generating a "raw field - standard field" mapping rule set;

[0082] Metadata governance module 38: Record the business meaning, data lineage, and lifecycle status (active / archived / obsolete) of the fields; data lineage refers to the tracking and recording of the entire process of data from generation and flow to final consumption.

[0083] Interface Collaboration Module 35: Dynamically generates multi-data source field mapping tables, supporting automatic concatenation and format conversion of interface call results;

[0084] Cache control module 36: Based on Redis / MongoDB caching strategies, it supports cache writing and reading by interface, query conditions, and time threshold (TTL);

[0085] Load balancing module 37: Based on weight allocation and round-robin algorithm for multi-data source scheduling, combined with the Hystrix framework to implement service circuit breaking and degradation.

[0086] In this embodiment, the calculation of field semantic similarity in the intelligent label recognition module 33 includes:

[0087] The BERT model is used to vectorize field names, and the cosine similarity with the standard field dictionary is calculated using the following formula.

[0088] ;

[0089] Cosine similarity is denoted as A is the field to be compared, and B is a field in the standard field dictionary;

[0090] Set a threshold (e.g., ≥0.8) to determine whether a field is a synonym. If the value is below the threshold, a manual review process will be triggered.

[0091] The cache update strategy in the cache control module 36 of this application embodiment includes:

[0092] Define the cache key as "MD5(interface ID + query parameters)" and use the LRU eviction policy. The Redis update logic is as follows:

[0093] public String getRedisData(String serviceCode, String request, longttl) {

[0094] String key = MD5Util.getMd5(serviceCode + request);

[0095] if (RedisUtils.hasKey(key)){

[0096] return (String) RedisUtils.get(key);

[0097] }

[0098] String data = callThirdPartyApi(serviceCode, request);

[0099] RedisUtils.setExpire(key, data, ttl); / / TTL is determined by the interface configuration.

[0100] return data;

[0101] }

[0102] The unified label data processing system based on a multi-source data environment provided in this application embodiment can improve field mapping configuration efficiency by 80%, achieve semantic consistency accuracy of ≥95%, reduce the cost of third-party interface calls by more than 30% through cache reuse, and enable system availability to reach 99.9% through a service circuit breaker mechanism.

[0103] The unified tag data processing system based on a multi-source data environment provided in this application includes a data access module 31, a unified tag dictionary module 32, an intelligent tag recognition module 33, a field standardization module 34, a metadata governance module 38, an interface collaboration module 35, a cache control module 36, and a load balancing module 37. It calculates the semantic similarity between field names and standard field dictionaries using a pre-trained semantic model (such as BERT), and generates mapping rules using a rule engine to achieve intelligent matching and standardization of fields. It schedules multiple data source interfaces based on weight allocation and round-robin algorithms, and integrates a circuit breaker mechanism to ensure service stability. It reduces the frequency of interface calls through differentiated caching strategies (such as Redis / MongoDB) and reuses historical query results to optimize costs.

[0104] The unified tag data processing system based on a multi-source data environment provided in this application can improve field mapping efficiency and semantic consistency, reduce manual intervention, and improve development efficiency; reduce the cost of third-party interface calls through caching and reuse; and enhance system fault tolerance and business continuity. It is suitable for scenarios such as financial lending and data platforms that need to interface with multi-source heterogeneous data, and has significant practical value and economic benefits.

[0105] The above description is merely a preferred embodiment of the present invention and does not limit the implementation and protection scope of the present invention. Those skilled in the art should realize that any equivalent substitutions and obvious changes made based on the description and illustrations of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for data processing based on unified labels in a multi-source data environment, characterized in that, The method comprises the following steps: receiving a business request by calling an internal data service interface; matching the business request by a standard field dictionary to obtain service configuration information; querying the database to determine whether the service configuration information exists; if the service configuration information does not exist in the database, querying a third-party source data interface service, caching the queried data to the database through the third-party source data interface service, setting a cache time, and returning the data; wherein the caching of the data to the database through the third-party source data interface service comprises processing the data through a pre-configured load balancing and fuse strategy; performing data cleaning and conversion according to the internal and third-party source data association relationship of the service configuration information to obtain target data; storing the target data according to metadata and caching the target data; Before the step of receiving a business request by calling an internal data service interface, the method further comprises the following steps: configuring a unified label field, wherein a standard field dictionary is configured to define a standard label field of internal unified semantics of an organization, and the standard field dictionary includes field code, field name, field description, field explanation, data type, and data sample; configuring an internal data service, wherein the internal data service is an internal data output specification of an organization, and the internal data service includes data service code, standard output data structure, and internal API interface calling service usage instruction; wherein the data service code includes various data service interface code values; the standard output data structure is a unified data structure for internal data service output, and the field name in the unified data structure uses a unified label code; configuring a third-party source data service, wherein the third-party source data service is an API interface service for connecting external data sources, and the third-party source data service includes a third-party interface URL and a third-party output data structure; configuring an internal and third-party source data association relationship, wherein the internal and third-party source data association relationship is to associate the internal data service with the third-party source data service, to establish a mapping relationship between the internal standard output data structure and the third-party output data structure, and to establish a field mapping relationship by first matching with the standard field dictionary, establishing a field mapping association if the matching score is greater than or equal to a threshold, entering a rule engine configuration if the matching score is lower than the threshold, establishing a mapping relationship through manual matching if no mapping relationship has been established, and sending the matching field to an intelligent label recognition module for model training after the mapping relationship is established; configuring service configuration parameters, wherein the service configuration parameters are data caching and load balancing of the internal data service, the data caching is used to configure a cache database and a cache time, and the load balancing is used to select round robin, weight, and master and backup. The processing of the data through the preconfigured load balancing and fusing strategy comprises: in the case that the load balancing comprises a polling mode, a weight mode and a master-slave mode, when the load balancing is the polling mode, the service request uses a first data source for querying, and when the service request is initiated again, a second data source is used for querying; when the load balancing is the weight mode, according to a preset weight, when the number of times of calling the first data source accounts for less than a preset value in the total number of times, the first data source is used for querying; when the number of times of calling the first data source accounts for more than the preset value in the total number of times, the second data source is used for querying; when the load balancing is the master-slave mode, the first data source is set as a master data source, and the second data source is set as a backup data source, the query of the service request is executed through the first data source, and when the first data source service is abnormal, the service request is sent to the second data source; The caching of the data to the database through the preconfigured load balancing and fusing strategy comprises: when the three-party source data service is continuously abnormal for a plurality of times within a preset time, an interface fusing is triggered, the request to the three-party source data service is stopped, and the three-party source data service is unshipped, and meanwhile, a three-party source data service abnormality warning is issued until the three-party source data service is recovered, and the interface service of the three-party source data service is shipped. The saving of the target data according to the metadata comprises: saving the target data according to the metadata, storing into a metadata table, and obtaining the target data of a metadata structure, wherein the target data of the metadata structure comprises: data source, use business scenario and query time of the target data.

2. The method of claim 1, wherein, The matching according to the service request through a standard field dictionary to obtain service configuration information comprises: Feature extraction of the service request based on natural language processing and a pre-trained semantic model to obtain a field name, annotation and sample value of to-be-processed data in the service request; Matching the field name, annotation and sample value of the to-be-processed data with the standard field dictionary to obtain the service configuration information.

3. The method of claim 1, wherein, After the service configuration information is queried from the database, the method further comprises: If the service configuration information exists in the database, corresponding cached data is returned.

4. The method of claim 1, wherein, The three-party source data interface service is an interface for connecting three-party API, assembling calling data, calling process and data encryption and decryption.

5. The data processing method based on unified labels in a multi-source data environment according to claim 1, characterized in that, The metadata in the metadata table is used to record business meaning, data bloodline and life cycle state of a field; wherein the life cycle state comprises: active, archived or abandoned; and the data bloodline refers to tracking and recording of a whole process from generation to finalization of data.

6. A unified tag based data processing system based on multi-source data environment characterized in that, The data processing method based on unified labels in a multi-source data environment according to any one of claims 1 to 5, comprising: A data access module is configured to receive multi-source data and extract field name, value and structure information of the multi-source data. A unified label dictionary module is configured to store a standard field dictionary, which includes a storage field code, a standard name, a business semantic, a data type, and a sample value; An intelligent label recognition module is configured to match field names, values, and structure information of the multi-source data with the standard field dictionary according to natural language processing and a pre-trained semantic model, and obtain a matching result; A field standardization module is configured to generate a mapping rule set of the matching result through an internal rule engine and an artificial checking interface; An interface coordination module is configured to automatically splice and format convert according to the mapping rule set through dynamically generating a multi-source field mapping table, and obtain a calling instruction; A cache control module is configured to execute calling according to the calling instruction, and obtain a calling result; A load balancing module is configured to perform load balancing according to the calling result, and obtain a processing result; A metadata management module is configured to record a whole process of a field.

Images