Unified label data processing method and system based on multi-source data environment

Abstract

The invention relates to a data processing method and system of a unified label based on a multi-source data environment. The method comprises the following steps: calling an internal data service interface to match a received service request through a standard field dictionary to obtain service configuration information; querying whether service configuration information exists or not through a database; querying a three-party source data interface service, and processing the queried data through the three-party source data interface service according to a pre-configured load balancing and fusing strategy; performing data cleaning conversion according to the association relationship between the interior of the service configuration information and the three-party source data to obtain target data; the method has the advantages that the field mapping efficiency and semantic consistency are improved, manual intervention is reduced, the development efficiency is improved, the three-party interface calling cost is reduced through cache multiplexing, and the system fault tolerance and service continuity are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology applications, and in particular to a data processing method and system based on unified tags in a multi-source data environment. Background Art

[0002] In the financial credit business, companies need to connect to third-party data sources such as credit reporting agencies, social security platforms, and e-commerce platforms to obtain user credit scores, income certificates, consumer behavior, and other data. However, the interfaces between different data sources have the following problems:

[0003] 1. Field naming differences: The same semantic field is named differently in different interfaces (such as "CreditScore" and "Credit Score");

[0004] 2. Structural heterogeneity: The return structure of similar data is different (such as different JSON nesting levels and field grouping methods);

[0005] 3. Service stability: There is a lack of a circuit breaker and downgrade mechanism when a data source call fails, which can easily lead to cascading system failures.

[0006] 4. Cost control: High-frequency calls lead to a surge in interface costs and a lack of cache reuse mechanism.

[0007] Existing technologies mostly rely on manual configuration of field mapping rules, which has drawbacks such as low efficiency, high maintenance costs, and poor scalability. This invention proposes a solution that integrates semantic recognition and automated governance.

[0008] Currently, related technologies rely on manual configuration of field mapping rules, resulting in low efficiency, high maintenance costs, and poor scalability. They are unable to solve the problems of multi-source interface field naming differences, structural heterogeneity, service instability, and high-frequency call costs. No effective solution has been proposed yet. Summary of the Invention

[0009] The purpose of the present invention is to address the deficiencies in the existing technology and provide a data processing method and system based on unified labels in a multi-source data environment to solve the technical problems in the related technology that rely on manually configured field mapping rules, resulting in low efficiency, high maintenance costs, poor scalability, and inability to solve the naming differences of multi-source interface fields, structural heterogeneity, service instability and high-frequency call costs.

[0010] To achieve the above object, the technical solution adopted by the present invention is:

[0011] The application provides a data processing method based on unified label of multi-source data environment, comprising: receiving a business request by calling an internal data service interface; obtaining service configuration information by matching through a standard field dictionary according to the business request; querying whether the service configuration information exists in a database; if the service configuration information does not exist in the database, querying a three-party source data interface service, caching the queried data to the database through the three-party source data interface service, setting a cache time, and returning the data; wherein the caching the queried data to the database through the three-party source data interface service comprises: processing the data through a pre-configured load balancing and fuse strategy; obtaining target data by data cleaning and conversion according to the internal and three-party source data association relationship of the service configuration information; saving the target data according to metadata, and caching the target data.

[0012] Optionally, before receiving the business request by calling the internal data service interface, the method further comprises: configuring a unified label field, wherein the standard field dictionary is configured to define the standard label field of the unified semantics of the organization, and the standard field dictionary comprises: field code, field name, field description, field explanation, data type and data sample; configuring an internal data service, wherein the internal data service is an organization internal data output specification, and the internal data service comprises: data service code, standard output data structure and internal API interface service usage instruction; wherein the data service code comprises: various data service interface code values; the standard output data structure is a unified data structure output by the internal data service, and the field name in the unified data structure uses a unified label code; configuring a three-party source data service, wherein the three-party source data service is an API interface service connected to an external data source, and the three-party source data service comprises: a three-party interface URL and a three-party output data structure; configuring an internal and three-party source data association relationship, wherein the internal and three-party source data association relationship is to associate the internal data service with the three-party source data service, the internal standard output data structure and the three-party output data structure establish a mapping relationship, the field mapping relationship is established by first matching with the standard field dictionary, if the matching score is greater than or equal to a threshold value, the field mapping relationship is established; if the matching score is less than the threshold value, a rule engine configuration is entered, if no mapping relationship has been established, a mapping relationship is established by manual matching, and after the mapping relationship is established, the matching field is sent to an intelligent label recognition module for model training; configuring a service configuration parameter, wherein the service configuration parameter is a data cache and load balancing of the internal data service, the data cache is used for configuring a cache database and a cache time, and the load balancing is used for selecting round robin, weight and master-slave.

[0013] Optionally, the service configuration information is obtained by matching the business request with the standard field dictionary, including: performing feature extraction on the business request based on natural language processing and a pre-trained semantic model to obtain a field name, an annotation, and a sample value of the to-be-processed data in the business request; and matching the field name, the annotation, and the sample value of the to-be-processed data with the standard field dictionary to obtain the service configuration information.

[0014] Optionally, after determining whether the service configuration information exists in the database, the method further includes: if the service configuration information exists in the database, returning corresponding cache data.

[0015] Optionally, the three-party source data interface service is an interface for connecting a three-party API, and the three-party source data interface service assembles calling data, a calling process, and data encryption and decryption.

[0016] Optionally, processing the data by using the preconfigured load balancing and fuse strategies includes: in a case where the load balancing includes a round robin mode, a weight mode, and a master-slave mode, when the load balancing is in the round robin mode, the business request is queried by using a first data source, and when the business request is initiated again, the business request is queried by using a second data source; when the load balancing is in the weight mode, according to a preset weight, it is set that when a number of times of calling the first data source accounts for a total number of times is less than a preset value, the first data source is used for querying; when the number of times of calling the first data source accounts for the total number of times is greater than the preset value, the second data source is used for querying; and when the load balancing is in the master-slave mode, the first data source is set as a master data source, and the second data source is set as a backup data source, the business request is executed by using the first data source, and when the first data source is abnormal, the business request is sent to the second data source.

[0017] Further, optionally, processing the data by using the preconfigured load balancing and fuse strategies includes: when the three-party source data service is abnormal for a plurality of times continuously within a preset time, triggering interface fuse, stopping initiating a request to the three-party source data service, and deactivating the interface service of the three-party source data service, and simultaneously issuing a warning of the three-party source data service being abnormal until the three-party source data service is recovered and the interface service of the three-party source data service is activated.

[0018] Optionally, saving the target data according to the metadata includes: saving the target data according to the metadata and storing the target data in a metadata table to obtain target data of a metadata structure, wherein the target data of the metadata structure includes: a data source of the target data, a business scenario of use, and a query time.

[0019] Further, optionally, the metadata in the metadata table is used to record a business meaning of a field, data provenance, and a lifecycle state; wherein the lifecycle state includes: active, archived, or abandoned; and the data provenance refers to tracking and recording of a whole process of data from generation to finalization.

[0020] The application provides a data processing system based on unified labels of a multi-source data environment, comprising: a data access module configured to receive multi-source data and extract field names, values and structure information of the multi-source data; a unified label dictionary module configured to store a standard field dictionary, the standard field dictionary comprising: a stored field code, a standard name, a business semantic, a data type and a sample value; an intelligent label identification module configured to match the field names, values and structure information of the multi-source data with the standard field dictionary according to natural language processing and a pre-trained semantic model to obtain a matching result; a field standardization module configured to generate a mapping rule set of the matching result through an embedded rule engine and an artificial checking interface; an interface coordination module configured to automatically splice and format convert according to the mapping rule set through dynamically generating a multi-source field mapping table to obtain a calling instruction; a cache control module configured to execute the calling according to the calling instruction to obtain a calling result; a load balancing module configured to perform load balancing according to the calling result to obtain a processing result; and a metadata management module configured to record the whole process of the field.

[0021] The application adopts the above technical scheme, receives a business request through an internal data service interface; matches the business request through a standard field dictionary to obtain service configuration information; queries whether the service configuration information exists in a database; if the service configuration information does not exist in the database, queries a three-party source data interface service, caches the queried data to the database through the three-party source data interface service, sets a cache time, returns the data; wherein caching the queried data to the database through the three-party source data interface service comprises: processing the data through a pre-configured load balancing and fuse strategy; performing data cleaning and conversion according to an internal and three-party source data association relationship of the service configuration information to obtain target data; saving the target data according to metadata and caching the target data, compared with the prior art, has the following technical effects: improving field mapping efficiency and semantic consistency, reducing manual intervention, improving development efficiency, reducing three-party interface calling cost through cache reuse, and enhancing system fault tolerance and business continuity. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of a data processing method based on unified labels of a multi-source data environment according to embodiment one of the application;

[0023] Figure 2 is a schematic diagram of data management of a data processing method based on unified labels of a multi-source data environment according to embodiment one of the application;

[0024] Figure 3 is a schematic diagram of a data processing system based on unified labels of a multi-source data environment according to embodiment two of the application;

[0025] Figure 4 is a schematic diagram of a data processing system based on unified labels in a multi-source data environment according to the second embodiment of the present application in a data service configuration;

[0026] Figure 5 is a schematic diagram of a data processing system based on unified labels in a multi-source data environment according to the second embodiment of the present application in data governance. DETAILED DESCRIPTION

[0027] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is described and explained below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0028] Obviously, the drawings in the following description are only some examples or embodiments of the present application, and for those of ordinary skill in the art, the present application can be applied to other similar scenarios without creative labor on the basis of these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the present application, some designs, manufacturing or production changes based on the technical content disclosed in the present application are only routine technical means and should not be understood as insufficient disclosure of the present application.

[0029] In the present application, the phrase "embodiment" means that the specific features, structures or properties described in combination with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor is it an independent or alternative embodiment that is not mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in the present application can be combined with other embodiments without conflict.

[0030] Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by a person of ordinary skill in the technical field to which this application belongs. The words "a", "an", "a", "the" and the like used in this application do not indicate a limit on quantity and may indicate the singular or plural. The terms "include", "comprise", "have" and any variations thereof used in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or units (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units that are inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The words "multiple" / "several" used in this application refer to two or more. "And / or" describes the association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, or B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0031] Example 1

[0032] An exemplary embodiment of the present invention is as follows Figure 1 As shown, Figure 1 1 is a flow chart of a data processing method based on a unified label in a multi-source data environment according to a first embodiment of the present invention. The data processing method based on a unified label in a multi-source data environment provided by the embodiment of the present application includes:

[0033] Step S100, receiving a service request by calling an internal data service interface;

[0034] Optionally, before receiving the service request by calling the internal data service interface in step S100, the unified label data processing method based on a multi-source data environment provided in the embodiments of the present application further comprises: configuring a unified label field, wherein a standard field dictionary is configured to define a standard label field of internal unified semantics, and the standard field dictionary comprises: field code, field name, field description, field specification, data type, and data sample; configuring an internal data service, wherein the internal data service is an internal data output specification of an organization, and the internal data service comprises: data service code, standard output data structure, and internal API interface calling service usage specification; the data service code comprises: various data service interface code values; the standard output data structure is a unified data structure output by the internal data service, and the field name in the unified data structure uses a unified label code; configuring a three-party source data service, wherein the three-party source data service is an API interface service connected to an external data source, and the three-party source data service comprises: a three-party interface URL and a three-party output data structure; configuring an internal and three-party source data association relationship, wherein the internal and three-party source data association relationship is to associate the internal data service with the three-party source data service, the internal standard output data structure and the three-party output data structure establish a mapping relationship, the field mapping relationship is established by first matching with the standard field dictionary, if the matching score is greater than or equal to a threshold value, the field mapping association is established; if the matching score is less than the threshold value, a rule engine configuration is entered, if no mapping relationship has been established, a mapping relationship is established by manual matching, and after the mapping relationship is established, the matching field is sent to an intelligent label recognition module for model training; and configuring a service configuration parameter, wherein the service configuration parameter is a data cache and load balancing of the internal data service, the data cache is used to configure a cache database and a cache time, and the load balancing is used to select polling, weight, and master and backup.

[0035] Specifically, the unified label data processing method based on a multi-source data environment provided in the embodiments of the present application can be applied to the field of financial credit, and the unified label data processing method based on a multi-source data environment provided in the embodiments of the present application comprises two parts.

[0036] The first part is data service configuration

[0037] Before step S100 is executed, a data configuration function needs to be performed on a background management page, and the data service configuration comprises: an internal unified label dictionary, an internal data service, a three-party source data service, an internal and three-party source data association relationship, and a service configuration; wherein,

[0038] The unified label field is configured first, the internal unified label dictionary is a standard label dictionary of internal unified semantics of a company, and comprises: field code, field name, field description, field specification, data type, and data sample, and each business field is defined according to semantics to configure a unified internal label dictionary.

[0039] Configure internal data services, which are internal data output specifications of a company (i.e., an organization in the embodiment of the present application), and contain data service codes (different data service interface code values), standard output data structures (uniform data structures for internal data service output, field names in the data structures use uniform label codes, and semantics are uniform), and internal API interface calling services, so that internal personnel can understand the business meaning of each field of the output.

[0040] Configure three-party source data services, which are API interface services for interfacing external data sources, and contain three-party interface URLs and three-party output data structures, which specify which three-party source data interface to call.

[0041] Configure internal and external data service association relationships, which are associations between internal data services and three-party source data services, and mapping relationships between internal standard output data structures and three-party output data structures (for example, a field zs from an A credit investigation agency is mapped to an internal field creditScore). The field mapping relationship is established by first matching through an intelligent label recognition module, and if the matching score is greater than or equal to a threshold (such as ≥ 0.8), the field mapping association is established, and if the score is lower than the threshold, the rule engine configuration is entered. Finally, if the mapping relationship has not been established, manual matching is performed. After the mapping relationship is established, the matched fields are sent to the intelligent label recognition module for model training, to improve the success rate of the next matching.

[0042] Configure service configuration parameters, which are internal data service data caching and load balancing mechanisms. The caching mechanism can be configured to use a cache database (such as Redis / MongoDB) and a cache time. The load balancing mechanism can select round robin, weight, and primary / backup.

[0043] Second part, data governance

[0044] The embodiment of the present application provides internal data governance services after the completion of data service configuration, i.e., steps S100 to S110 are executed. In step S100, a business request is received by calling an internal data service interface.

[0045] A business department calls an internal data service interface, which is an API interface and a standard data structure output provided to internal departments of a company to call three-party source data interface services.

[0046] Step S102, according to the business request, matching is performed through a standard field dictionary to obtain service configuration information;

[0047] Optionally, the service configuration information obtained by matching the service request through the standard field dictionary in step S102 comprises: performing feature extraction on the service request based on natural language processing and a pre-trained semantic model to obtain a field name, an annotation, and a sample value of the to-be-processed data in the service request; and matching the field name, the annotation, and the sample value of the to-be-processed data with the standard field dictionary to obtain the service configuration information.

[0048] Specifically, a label is obtained using a standard field dictionary of internal unified semantics, relevant service configuration information is queried according to the internal data service, and then the configuration information is used to initiate a three-party source data query.

[0049] In step S104, it is determined whether the service configuration information exists in the database.

[0050] Optionally, after the determination of whether the service configuration information exists in the database in step S104, the method further comprises: if the service configuration information exists in the database, returning corresponding cached data.

[0051] Specifically, the data caching function is used before the three-party source data is queried. The data caching function is a function implementation of cache configuration for data configuration. According to the configuration, a corresponding cache library is used to save and query data. For example, getRedisData(String serviceCode, String request, long ttl) is used. Each time, it is determined whether the data exists in the redis database. If the data exists, the cached data is output. If the data does not exist, the three-party source data interface service is queried. The queried data is cached in the redis database with a cache time. Then, the data is output. That is, it is determined whether the service configuration information exists in the database. If the service configuration information exists, corresponding cached data is returned. If the service configuration information does not exist, step S106 is performed.

[0052] In step S106, if the service configuration information does not exist in the database, the three-party source data interface service is queried. The queried data is cached in the database through the three-party source data interface service, a cache time is set, and the data is returned. The caching of the queried data in the database through the three-party source data interface service comprises: processing the data through a pre-configured load balancing and a fuse strategy.

[0053] Optionally, the three-party source data interface service is an API interface connected to a third party. The three-party source data interface service assembles calling data, a calling process, and data encryption and decryption.

[0054] Optionally, the processing of the data through the preconfigured load balancing and fusing strategy includes: in the case that the load balancing includes a round robin mode, a weight mode and a master-slave mode, when the load balancing is the round robin mode, the service request is queried using the first data source, and when the service request is initiated again, the second data source is used for querying; when the load balancing is the weight mode, according to a preset weight, it is set that when the number of times of calling the first data source accounts for less than a preset value of the total number of times, the first data source is used for querying; when the number of times of calling the first data source accounts for more than the preset value of the total number of times, the second data source is used for querying; and when the load balancing is the master-slave mode, the first data source is set as a master data source, and the second data source is set as a backup data source, the query of the service request is executed through the first data source, and when the first data source service is abnormal, the service request is sent to the second data source.

[0055] Further, optionally, the processing of the data through the preconfigured load balancing and fusing strategy includes: when the three-party source data service is continuously abnormal for multiple times within a preset time, an interface fusing is triggered, the request to the three-party source data service is stopped, and the three-party source data service is unshipped, and at the same time, an abnormal warning of the three-party source data service is sent, until the three-party source data service is recovered, and the interface service of the three-party source data service is shipped.

[0056] Specifically, the three-party source data interface service in the embodiment of the application is an interface for connecting a three-party API, and is used for assembling calling data, a calling process and data encryption and decryption.

[0057] In the process of initiating the calling of the three-party source data interface service, load balancing and fusing strategy are also made according to the service configuration in the data service configuration; wherein,

[0058] The load balancing is a function implementation of the load balancing configuration of the data configuration function, and different modes are used according to the configuration strategy, including: round robin, weight and master-slave, wherein,

[0059] Round robin: the current request uses A data source, and the next request uses B data source (that is, in the embodiment of the application, when the load balancing is the round robin mode, the service request is queried using the first data source, and when the service request is initiated again, the second data source is used for querying).

[0060] Weight: A:3, B:7 (that is, the preset weight in the embodiment of the application), the number of times of calling the A data source accounts for less than or equal to 30% of the total number of times (that is, the preset value in the embodiment of the application), and the A data source is used when the number of times of calling the A data source accounts for more than 30% of the total number of times (that is, in the embodiment of the application, according to the preset weight, it is set that when the number of times of calling the first data source accounts for less than a preset value of the total number of times, the first data source is used for querying).

[0061] Master / standby: A master B standby, all normal requests are sent to A data source, when A service is abnormal, the request is automatically sent to B data source (that is, in the embodiment of the application, when the load balancing is in the master / standby mode, the first data source is set as the master data source, the second data source is set as the standby data source, the business request query is executed through the first data source, and when the first data source service is abnormal, the business request is sent to the second data source).

[0062] Fusing is that when the third-party source data service is continuously abnormal for a period of time, the interface is fused, no request is initiated to the third-party source data service, the third-party source data service is unshelved, and a third-party source data service abnormal warning is issued, and the interface service is shelved again after the third-party source data service recovers (that is, in the embodiment of the application, when the third-party source data service is continuously abnormal for a preset time, the interface is fused, the request to the third-party source data service is stopped, and the third-party source data service is unshelved, and a third-party source data service abnormal warning is issued, until the third-party source data service recovers, and the interface service of the third-party source data service is shelved).

[0063] In step S108, the target data is obtained by data cleaning and conversion according to the internal and third-party source data association relationship of the service configuration information.

[0064] Specifically, the output data is obtained by querying the third-party source data, and the data cleaning and conversion is performed according to the internal and third-party source data association relationship of the data service configuration information, that is, the target data in the embodiment of the application.

[0065] In step S110, the target data is saved according to the metadata, and the target data is cached.

[0066] Optionally, the saving of the target data according to the metadata in step S110 includes: saving the target data according to the metadata and storing the target data in a metadata table to obtain the target data of the metadata structure, wherein the target data of the metadata structure includes: data source of the target data, use business scenario, and query time.

[0067] Further, optionally, the metadata in the metadata table is used to record the business meaning of the field, the data bloodline, and the life cycle state; wherein the life cycle state includes: active, archived or abandoned; and the data bloodline refers to the tracking and recording of the whole process from the generation of data to the final flow.

[0068] Specifically, after conversion into the standard output data structure, metadata is saved based on this data query. Standard source data metadata storage saves the data for each request, records the data source, usage business scenario, and query time, and saves the metadata into a table to facilitate tracking of data sources, processing methods, and data analysis.

[0069] After the metadata is saved, the standard output data is cached using the cache function, and then the standard output data is output to the business department.

[0070] In summary, the data processing method based on unified labels in a multi-source data environment provided by the embodiment of the present application unifies the field identification of the entire platform to reduce communication costs and improve data governance and analysis capabilities. Intelligent label recognition reduces the manual error rate and labor costs. Cache reuse improves system performance and reduces the cost of third-party calls. Load balancing and circuit breaker mechanisms enhance system fault tolerance and business continuity. The work of connecting to third-party source data interfaces is simpler and more convenient, reducing development workload.

[0071] like Figure 2 As shown, Figure 2 This is a schematic diagram of a data processing method based on unified labels in a multi-source data environment in data governance according to embodiment 1 of the present invention. The data processing method based on unified labels in a multi-source data environment provided by the embodiment of the present application is specifically as follows: receive data query, query data configuration, determine whether there is a cache, and if so, directly return the standard data result; if not, obtain the calling three-party interface configuration, and select one of the three-party interfaces A, B, and C through load balancing and circuit breaker strategies. If the standard input parameter is converted to A three-party input parameter, call the A three-party interface to obtain the A three-party output parameter to be converted into the standard output parameter; if the standard input parameter is converted to B three-party input parameter, call the B three-party interface to obtain the B three-party output parameter to be converted into the standard output parameter; if the standard input parameter is converted to C three-party input parameter, call the C three-party interface to obtain the C three-party output parameter to be converted into the standard output parameter; convert the A three-party output parameter into the standard output parameter, convert the B three-party output parameter into the standard output parameter, or convert the C three-party output parameter into the standard output parameter for data warehousing, so that the standard data result can be returned through subsequent cached data processing.

[0072] Among them, there are differences in the usage scenarios of load balancing in the embodiments of the present application. For example, to query the GPS latitude and longitude location, three data sources such as A degree, B degree, and C ladder can all be queried, and load balancing determines which data source to use to query the location.

[0073] The application adopts the above technical scheme, receives a service request through calling an internal data service interface; obtains service configuration information through matching according to the service request through a standard field dictionary; queries whether the service configuration information exists in a database; if the service configuration information does not exist in the database, queries a three-party source data interface service, caches the queried data to the database through the three-party source data interface service, sets a cache time, returns data; wherein caching the queried data to the database through the three-party source data interface service comprises: processing the data through a pre-configured load balancing and fuse strategy; obtaining target data through data cleaning and conversion according to the internal and three-party source data association relationship of the service configuration information; saving the target data according to metadata and caching the target data, compared with the prior art, has the following technical effects: improving field mapping efficiency and semantic consistency, reducing manual intervention, improving development efficiency, reducing three-party interface calling cost through cache reuse, and enhancing system fault tolerance and business continuity.

[0074] Embodiment 2

[0075] An illustrative embodiment of the application, as shown in Figure 3 , Figure 3 is a schematic diagram of a unified label data processing system based on a multi-source data environment according to Embodiment Two of the application, applied to the unified label data processing method based on a multi-source data environment in Embodiment One, the unified label data processing system based on a multi-source data environment provided by the application embodiment comprises:

[0076] A data access module 31 is configured to receive multi-source data and extract field names, values and structure information of the multi-source data; a unified label dictionary module 32 is configured to store a standard field dictionary, the standard field dictionary comprising: stored field codes, standard names, business semantics, data types and sample values; an intelligent label identification module 33 is configured to match the field names, values and structure information of the multi-source data with the standard field dictionary according to natural language processing and a pre-trained semantic model, and obtain a matching result; a field standardization module 34 is configured to generate a mapping rule set of the matching result through a built-in rule engine and a manual checking interface; an interface coordination module 35 is configured to generate a multi-source field mapping table through dynamic generation, automatically splice and format convert according to the mapping rule set, and obtain a calling instruction; a cache control module 36 is configured to execute calling according to the calling instruction and obtain a calling result; a load balancing module 37 is configured to execute load balancing according to the calling result and obtain a processing result; and a metadata management module 38 is configured to record the whole process of the field.

[0077] Specifically, the data processing system based on unified labels in a multi-source data environment according to the embodiment 1 of the present application includes two parts, a first part, data service configuration, and a second part, data governance. Figure 4 and Figure 5 as shown, Figure 4 is a schematic diagram of the data processing system based on unified labels in a multi-source data environment according to the embodiment 2 of the present application in data service configuration; Figure 5 is a schematic diagram of the data processing system based on unified labels in a multi-source data environment according to the embodiment 2 of the present application in data governance;

[0078] In the embodiment of the present application, the data access module 31 supports multi-protocol access such as HTTP / API / message queue, and parses heterogeneous data formats such as JSON / XML;

[0079] The unified label dictionary module 32 defines an internal standard field dictionary according to business requirements, and the dictionary includes field coding, field name, field description, field explanation, data type, and data sample;

[0080] The intelligent label recognition module 33 extracts features from original field names, annotations, and sample values based on natural language processing (NLP) and pre-trained semantic models, and matches the internal standard field dictionary;

[0081] The field standardization module 34 has a built-in rule engine (regular expression, decision tree) and an artificial verification interface, and generates a mapping rule set of “original field-standard field”;

[0082] The metadata governance module 38 records the business meaning, data bloodline, and life cycle state (active / archived / obsolete) of the field, and the data bloodline refers to the tracking and recording of the whole process from the generation, flow, to the final consumption of data.

[0083] The interface coordination module 35 dynamically generates a multi-data source field mapping table, and supports automatic splicing and format conversion of interface call results;

[0084] The cache control module 36 supports cache writing and reading according to interfaces, query conditions, and time thresholds (TTL) based on Redis / MongoDB cache strategies;

[0085] The load balancing and fusing module 37 combines the Hystrix framework to realize service fusing and degradation based on multi-data source scheduling of weight allocation and polling algorithms.

[0086] In the embodiment of the present application, the field semantic similarity calculation in the intelligent label recognition module 33 includes:

[0087] The BERT model is used to vectorize the field names and calculate the cosine similarity with the standard field dictionary. The formula is as follows

[0088]

[0089] Among them, the cosine similarity is recorded as Similarity(A, B), A is the field to be compared, and B is the field in the standard field dictionary;

[0090] Set a threshold (such as ≥0.8) to determine whether it is a synonymous field. When it is lower than the threshold, the manual review process is triggered.

[0091] The cache update strategy in the cache control module 36 in the embodiment of the present application includes:

[0092] Define the cache key as "MD5 (interface ID + query parameters)" and adopt the LRU elimination strategy. The Redis update logic is as follows:

[0093]

[0094] The data processing system based on unified tags in a multi-source data environment provided by the embodiment of the present application can achieve an 80% improvement in field mapping configuration efficiency and a semantic consistency accuracy rate of ≥95%; reduce the cost of third-party interface calls by more than 30% through cache reuse; and the service circuit breaker mechanism enables system availability to reach 99.9%.

[0095] The data processing system based on unified labels in a multi-source data environment provided in the embodiment of the present application includes a data access module 31, a unified label dictionary module 32, an intelligent label identification module 33, a field standardization module 34, a metadata governance module 38, an interface collaboration module 35, a cache control module 36 and a load balancing and fuse module 37. It calculates the semantic similarity between the field name and the standard field dictionary through a pre-trained semantic model (such as BERT), and generates mapping rules in combination with a rule engine to achieve intelligent matching and standardization of fields; schedules multiple data source interfaces based on weight distribution and polling algorithms, and integrates a fuse mechanism to ensure service stability; reduces the frequency of interface calls through differentiated caching strategies (such as Redis / MongoDB), and reuses historical query results to optimize costs.

[0096] The data processing system based on unified tags in a multi-source data environment, provided in this application embodiment, can improve field mapping efficiency and semantic consistency, reduce manual intervention, and enhance development efficiency; reduce the cost of third-party interface calls through cache reuse; and enhance system fault tolerance and business continuity. It is suitable for scenarios such as financial credit and data middleware that require the integration of multi-source heterogeneous data, and has significant practical value and economic benefits.

[0097] The above merely describes preferred embodiments of the present application, and is not intended to limit the implementation and protection scope of the present application. Those skilled in the art should be able to understand that any equivalent substitutions and obvious changes made according to the present application description and drawings should be included in the protection scope of the present application.

Claims

1. A data processing method based on unified labels in a multi-source data environment, characterized in that: include: Receive business requests by calling the internal data service interface; Matching the service request with a standard field dictionary to obtain service configuration information; Querying the database to determine whether the service configuration information exists; If the service configuration information does not exist in the database, querying a third-party source data interface service, caching the queried data in the database through the third-party source data interface service, setting a cache time, and returning the data; wherein caching the queried data in the database through the third-party source data interface service includes: processing the data through a pre-configured load balancing and circuit breaker strategy; Perform data cleansing and conversion based on the internal relationship between the service configuration information and the third-party source data to obtain the target data; The target data is saved according to the metadata, and the target data is cached.

2. The data processing method based on unified label in a multi-source data environment according to claim 1 is characterized in that: Before receiving the service request by calling the internal data service interface, the method further includes: Configure unified label fields, where standard label fields with unified semantics within the organization are defined by configuring a standard field dictionary. The standard field dictionary contains: field code, field name, field description, field explanation, data type, and data sample; Configure internal data services. The internal data services are the organization's internal data output specifications. The internal data services include: data service codes, standard output data structures, and instructions for using internal API interface calls. The data service codes include various data service interface code values. The standard output data structure is a unified data structure for internal data service output, and field names in the unified data structure are encoded using unified tags. Configure a third-party source data service, wherein the third-party source data service is an API interface service for connecting to an external data source, and the third-party source data service includes: a third-party interface URL and a third-party output data structure; Configuring the association relationship between internal and third-party source data. The association relationship between internal and third-party source data is to associate the internal data service with the third-party source data service, establish a mapping relationship between the internal standard output data structure and the third-party output data structure, and establish the field mapping relationship by first matching with the standard field dictionary. If the matching score is greater than or equal to the threshold, the field mapping association is established; if the matching score is lower than the threshold, the rule engine configuration is entered. If the mapping relationship has not been established, the mapping relationship is established through manual matching. After the mapping relationship is established, the matching fields are sent to the smart tag recognition module for model training; Configure service configuration parameters, which are used to configure data cache and load balancing of internal data services. The data cache is used to configure the use of cache database and cache time; the load balancing is used to select polling, weight and master / slave.

3. The data processing method based on unified label in a multi-source data environment according to claim 1 is characterized in that: The service configuration information obtained by matching the service request through the standard field dictionary includes: Perform feature extraction on the business request based on natural language processing and a pre-trained semantic model to obtain field names, comments, and sample values ​​of the data to be processed in the business request; The field name, comment, and sample value of the data to be processed are matched with the standard field dictionary to obtain the service configuration information.

4. The data processing method based on unified label in a multi-source data environment according to claim 1, characterized in that: After querying the database to determine whether the service configuration information exists, the method further includes: If the service configuration information in the database exists, the corresponding cache data is returned.

5. The data processing method based on unified label in a multi-source data environment according to claim 1, characterized in that: The third-party source data interface service is to connect to the third-party API interface, assemble call data, call process and data encryption and decryption.

6. The data processing method based on unified label in a multi-source data environment according to claim 1, characterized in that: The processing of the data by using the preconfigured load balancing and circuit breaker strategies includes: In the case where the load balancing includes a polling mode, a weighted mode, and an active / standby mode, when the load balancing is in the polling mode, the service request is queried using the first data source, and when the service request is initiated again, the service request is queried using the second data source; When the load balancing is in the weighted mode, according to the preset weight, when the number of calls of the first data source accounts for less than a preset value of the total number of calls, the first data source is used for query; when the number of calls of the first data source accounts for more than a preset value of the total number of calls, the second data source is used for query; When the load balancing is in the master-slave mode, the first data source is set as the master data source, and the second data source is set as the slave data source. The query of the business request is executed through the first data source. When the service of the first data source is abnormal, the business request is sent to the second data source.

7. The data processing method based on unified label in a multi-source data environment according to claim 6, characterized in that: The caching of the data in the database through the preconfigured load balancing and circuit breaker strategy includes: When the third-party source data service is abnormal multiple times in a preset time, the interface fuse is triggered, requests to the third-party source data service are stopped, the third-party source data service is removed from the shelves, and a third-party source data service abnormality warning is issued at the same time until the third-party source data service is restored and the interface service of the third-party source data service is put on the shelves.

8. The data processing method based on unified label in a multi-source data environment according to claim 1, characterized in that: The storing of the target data according to the metadata includes: The target data is saved according to the metadata and stored in a metadata table to obtain the target data in a metadata structure, wherein the target data in the metadata structure includes: data source, usage business scenario, and query time of the target data.

9. The data processing method based on unified label in a multi-source data environment according to claim 8, characterized in that: The metadata in the metadata table is used to record the business meaning, data lineage, and lifecycle status of the field; wherein the lifecycle status includes: active, archived, or discarded; the data lineage refers to the tracking and recording of the entire process from data generation to flow to the final end.

10. A data processing system based on unified labels in a multi-source data environment, characterized in that: include: A data access module is used to receive multi-source data and extract field names, values ​​and structure information of the multi-source data; A unified tag dictionary module is used to store a standard field dictionary, wherein the standard field dictionary includes: a storage field code, a standard name, business semantics, a data type, and a sample value; An intelligent tag recognition module is used to match the field names, values ​​and structure information of the multi-source data with the standard field dictionary based on natural language processing and pre-trained semantic models to obtain matching results; A field standardization module is used to generate a mapping rule set for the matching results through a built-in rule engine and a manual verification interface; The interface collaboration module is used to dynamically generate a multi-source field mapping table, automatically splice and convert the format according to the mapping rule set, and obtain a call instruction; A cache control module, configured to execute the call according to the call instruction and obtain the call result; A load balancing module is used to perform load balancing according to the call result and obtain a processing result; Metadata governance module, used to record the entire process of fields.

Images