External data management system and method, electronic equipment and medium

Abstract

The invention provides an external data management system and method, electronic equipment and a medium. The external data management system comprises a data access and integration module, a data storage module and a data query and analysis module, the data access and integration module acquires external data of different external data sources by adopting a preset acquisition standard, and cleans and integrates the external data; the data storage module stores the integrated external data and carries out structuring and standardization processing on the external data; and the data query and analysis module retrieves, queries and analyzes the external data of the data storage module based on the data demand of the user. The external data can be managed in a unified mode, and the use efficiency of the external data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data management, and in particular to an external data management system, method, electronic equipment and medium. Background Art

[0002] With the development of economy and the progress of society, the credit status of enterprises and individuals has been paid more and more attention. For financial institutions, data is generally divided into internal data and external data. Compared with internal data, external data has a larger volume, more sources and more complex structure. Due to the introduction of external data by multiple departments, there are many problems such as decentralized data management, unintegrated data sharing, insufficient utilization of data asset value, and difficulty in internal metering and billing. At the same time, the indicators and standards of external data are inconsistent, the quality is difficult to evaluate, the access requires customized research and development, the service lacks standards, and the internal and external cost verification is difficult. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an external data management system, method, electronic device and medium, which can uniformly manage external data and improve the use efficiency of external data.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] In the first aspect, the present invention provides an external data management system, including: a data access and integration module, a data storage module and a data query and analysis module; the data access and integration module adopts a preset collection standard to obtain external data from different external data sources, and cleans and integrates the external data; the data storage module stores the integrated external data, and structures and standardizes the external data; the data query and analysis module retrieves, queries and analyzes the external data of the data storage module based on the user's data needs.

[0006] Optionally, the data access and integration module includes: a data source management unit, which is used for external data source management and establishing an external data lifecycle management system; wherein, the external data lifecycle management system includes: when external data is introduced, the provider of external data is approved according to standardized approval processes and document templates, and the external data is introduced into the process for merging according to data requirements; when external data is used, the user's data requirements are reviewed according to pre-set review principles, and external data is uniformly obtained from the data storage module after the review is passed; when external data is evaluated, the external data is evaluated according to a pre-set evaluation template; wherein, data evaluation includes at least data quality monitoring, service evaluation, application effect evaluation, and provider evaluation.

[0007] Optionally, the data storage module includes: an external data storage layer, which is used to cache external data using a Redis cluster; an external data processing layer, which is used to perform structured parsing on received external data messages, store the parsed external data in the external data storage layer, and perform standardized conversion on external data message fields; an external data presentation layer, which is used to view basic information and called information of external data, and verify external data according to pre-set verification rules.

[0008] Optionally, the data query and analysis module includes: a unified data service interface for different data sources, which is used for querying and retrieving external data; a call service management unit, which is used to provide call services to external data sources, as well as caller service interface management, call detail management, call billing management, statistical analysis management, and batch call management; a derived indicator management unit, which is used for message structure management, offline message management, indicator template management, derived indicator management, derived indicator calculation and monitoring analysis; a data analysis unit, including a variety of data analysis tools and data analysis models, which is used to perform data analysis according to user data requirements.

[0009] Optionally, it also includes: a data security and authority management module, which is used to use data encryption algorithms and security protocols for data transmission, storage and processing, and to configure user data access rights.

[0010] Optionally, it also includes: a data export and sharing module, which is used to export external data according to a preset template.

[0011] Optionally, it also includes: a system maintenance and optimization module, which is used to monitor the system operating status and perform system operation, maintenance and management based on the system operating status.

[0012] In a second aspect, the present invention provides an external data management method, which is applied to any external data management system provided in the first aspect, including: obtaining external data from different external data sources through a data access and integration module using a preset collection standard, and cleaning and integrating the external data; storing the integrated external data through a data storage module, and structuring and standardizing the external data; retrieving, querying and analyzing the external data of the data storage module based on the user's data requirements through a data query and analysis module.

[0013] In a third aspect, the present invention provides an electronic device comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the method provided in the second aspect above.

[0014] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method provided in the second aspect are executed.

[0015] The present invention brings the following beneficial effects:

[0016] The above-mentioned external data management system, method, electronic device and medium provided by the present invention include: a data access and integration module, a data storage module and a data query and analysis module; the data access and integration module adopts a preset collection standard to obtain external data from different external data sources, and cleans and integrates the external data; the data storage module stores the integrated external data, and performs structured and standardized processing on the external data; the data query and analysis module retrieves, queries and performs data analysis on the external data of the data storage module based on the user's data needs. The above-mentioned system establishes unified external data collection standards and specifications through the data access and integration module, obtains external data from different external data sources, and integrates the external data, and then performs structured and standardized processing on the acquired external data through the data storage module; finally, the external data of the data storage module can be retrieved, queried and analyzed according to the user's data needs, so as to realize unified management of external data, ensure the integrity and consistency of data, and improve the efficiency of using external data.

[0017] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0018] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the preferred embodiments are specifically mentioned below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A schematic diagram of the structure of an external data management system provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of the structure of another external data management system provided by an embodiment of the present invention;

[0022] Figure 3 A flowchart of an external data management method provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] At present, although the existing external data management platforms have met the market demand to a certain extent, they still have deficiencies and defects in data security, data quality, system integration, data processing and analysis capabilities, costs, and user experience. The main points are as follows:

[0026] (1) High development cost: Existing similar products have high construction costs, data cannot be shared or negotiated in a unified manner, and the cost is high. As the number of interface calls continues to increase, the platform's computing resources are insufficient, and the platform will encounter performance bottlenecks when large-scale concurrent calls are made by downstream systems;

[0027] (2) Long construction cycle: It is difficult to achieve cost optimization and reconciliation with existing similar products. The cost settlement cycle is long, and the external data interface format is different. Each application within the organization needs to be customized and developed according to the interface format, which cannot be used quickly.

[0028] (3) Slow data access: There are many external data channels, data standards of different manufacturers are not unified, data storage is chaotic, data is uneven, and existing similar products are difficult to achieve fast access and unified management;

[0029] (4) Low service value: Financial institutions’ management of external data access is mostly non-standard, non-systematic, and decentralized. External data has quality issues, such as missing, duplicated, or inconsistent data formats. Existing products are not centrally managed, so it is impossible to achieve unified storage, data reuse, and deeper mining of data value.

[0030] (5) Difficulty in quality assessment: Financial institutions often lack the ability to verify the accuracy of data calls and are unable to objectively evaluate external data vendors. Existing products cannot subjectively evaluate data application services, resulting in uneven quality of institutional data services.

[0031] (6) Uncontrollable operating costs: Existing products lack a unified operating system. Multiple business systems have diverse and changing data requirements, resulting in repeated purchases and calls, making internal metering and billing difficult, and making it impossible to independently calculate business costs.

[0032] Based on this, the embodiments of the present invention provide an external data management system, method, electronic device and medium, which can uniformly manage external data and improve the use efficiency of external data.

[0033] To facilitate understanding of this embodiment, firstly, an external data management system disclosed in an embodiment of the present invention is described in detail, see Figure 1 The structural diagram of an external data management system shown in the figure illustrates that the system mainly includes the following parts: a data access and integration module 10, a data storage module 20 and a data query and analysis module 30.

[0034] The data access and integration module 10 uses a preset collection standard to obtain external data from different external data sources, and cleans and integrates the external data.

[0035] In one embodiment, the data access and integration module 10 first receives external data from different external data sources, which may be stored in files or databases of various formats and standards; then cleans and integrates the received external data so as to integrate it into a unified external data management system. In this process, the integrity, accuracy and security of the data need to be verified to ensure that only data that meets the requirements is accessed.

[0036] In specific implementation, the data access and integration module 10 can be configured to access external data sources, obtain data, and perform preliminary cleaning and verification of the accessed external data to ensure data quality. The module supports multiple data collection methods, such as API interface, FTP, database connection, etc., supports multiple message format access, and then performs unified message format forwarding services. In the process of access data analysis, the message field can be disassembled, data analysis mapping, etc., and further performs authentication management and unified key management on the accessed data, and publishes the interface standards and specifications of financial institutions.

[0037] The data storage module 20 stores the integrated external data and performs structured and standardized processing on the external data.

[0038] In one embodiment, the cleaned and integrated external data can be stored in the data storage module 20 of the system for subsequent analysis and processing. The module can also be responsible for maintaining the integrity and consistency of the data to ensure that the data is not tampered with or damaged during the storage process. At the same time, it also provides data backup and recovery functions to prevent data loss or damage.

[0039] The data query and analysis module 30 retrieves, queries and analyzes the external data of the data storage module based on the user's data requirements.

[0040] In one embodiment, the user can retrieve and query the stored external data through the query interface provided by the data query and analysis module 30. The embodiment of the present invention supports complex query conditions and advanced search functions to meet the different data needs of users. The user can also use the provided analysis tools to analyze and visualize the data to discover the value and rules of the data.

[0041] The above-mentioned external data management system provided by the present invention establishes unified external data collection standards and specifications through the data access and integration module, obtains external data from different external data sources, integrates the external data, and then structures and standardizes the obtained external data through the data storage module; finally, the external data of the data storage module can be retrieved, queried and analyzed according to the user's data needs, thereby realizing unified management of external data, ensuring the integrity and consistency of the data, and improving the efficiency of using external data.

[0042] In one embodiment, the data access and integration module 10 includes: a data source management unit, which is used for external data source management and establishing an external data lifecycle management system; wherein, the external data lifecycle management system includes: when external data is introduced, the provider of external data is approved according to standardized approval processes and document templates, and the external data is introduced into the process for merging according to data requirements; when external data is used, the user's data requirements are reviewed according to pre-set review principles, and external data is uniformly obtained from the data storage module after the review is passed; when external data is evaluated, the external data is evaluated according to a pre-set evaluation template; wherein data evaluation at least includes data quality monitoring, service evaluation, application effect evaluation, and provider evaluation.

[0043] In specific implementation, the data source management unit provides external data source management, call details, call billing, statistical analysis, batch call and other capabilities. Supports standardized field management, pre- / post-ETL processor configuration, Mock baffle / single / batch test multi-dimensional billing methods. The interface name, interface description, interface path, interface title, interface request method (GET / POST), interface version number, interface source code, interface parameters, interface return field, interface example, interface error code, etc. can be maintained and edited online. In terms of development services and procurement management: realize interface configuration, support interface Mock testing, and provide efficient development testing; realize access to data including online calling, batch pulling, and localized deployment; realize process management from demand to procurement through the management end APP.

[0044] The interface fields of the data source management unit include at least: provider name, provider code, interface name, interface code, interface status, interface purpose, payment method, interface unit price, cooperation start date, cooperation end date, creation time, update time, creator, updater, enable / disable, details, modification, paging, etc. In the embodiment of the present invention, the following functions can be set to implement interface management, including: external provider management, configurable data access, data structured landing, current limiting mechanism, cache mechanism, fuse mechanism, routing configuration, and standard interface. Specifically including:

[0045] (1) Traffic parameter configuration:

[0046] Configure the threshold of the number of times the interface can be called within a period of time. If the system calls the interface and the threshold is exceeded, the system stops calling the interface. Multiple traffic parameter limits can be configured for one interface, but only one configuration can be configured for one time unit.

[0047] (2) Query parameter configuration:

[0048] If a single call to the interface exceeds the configured duration, it will be considered a call failure.

[0049] When a single call to the interface fails, the call is re-initiated according to the configured number of retries. If the retry threshold is exceeded, the call is terminated.

[0050] If the interface output parameter data needs to be cached, the return time of the call interface data is recorded as the cache start time, and the cache expiration time is: start time + validity period;

[0051] When the upstream system initiates a call query, it reads the cache library for the call interface input parameter to see if there is data corresponding to the call. If so and the data is not expired, the cached data is returned to the upstream system. Otherwise, it re-initiates a call to the provider interface to obtain the latest user data.

[0052] (3) Monitoring alarm parameter configuration:

[0053] For the interface, configure the indicators that need to be monitored within a certain period of time. If the monitoring indicators exceed the threshold, an early warning is issued to the alarm object. One interface can be configured with multiple alarms.

[0054] (4) Comprehensive monitoring and early warning of service operation health status:

[0055] Summarizes and displays key system operating conditions such as service statistics and abnormal situation warnings, and provides management predictions for abnormal situations. It also supports monitoring of core business indicators, and business personnel can configure alarm strategies as needed to obtain abnormal information and resolve them in the first place.

[0056] (5) Interface integration, data caching and traffic warning: The system supports interface integration, which shortens the internal system development cycle, lowers the technical threshold, and saves technology expenditures. The system supports data persistence, improves the data service response efficiency, and reduces the cost of external data services. The system provides an early warning reminder mechanism to help customers reserve time for business docking.

[0057] In the embodiment of the present invention, data access is completed through visual template configuration, and no technical personnel are required to develop it. At the same time, the whole process data is standardized and compliantly managed, which is more secure and controllable. Through the data source management unit, zero-coding access to external interfaces and parameter standardization can be achieved, development agility can be improved, an API data service market can be established, and data integrity, timeliness and accuracy can be ensured; after editing the interface, a series of operations such as online interface testing, online interface debugging, interface publishing, interface authority management, and interface online documentation viewing can also be performed.

[0058] Furthermore, the data source management unit is also used to establish an external data full life cycle management system and optimize the management process. The external data full life cycle management system includes the following processes such as external data introduction, external data use, and external data evaluation:

[0059] (1) When introducing external data, it is necessary to regularly collect requirements, uniformly approve budgets, and uniformly apply for fees, and control costs in accordance with the principles of "minimum necessary" and "no duplication and no omissions". Standardize the shortlisted approval and data templates of external data providers, and merge the introduction process to improve the approval efficiency of external data and coordinate the business process. At the same time, the business information is synchronously entered into the external data management system to facilitate the billing and apportionment functions based on the amount of external data calls in the later stage. Data introduction should ensure unified acceptance and integration of data requirements, unified budget management and procurement, and strict control of provider access and data access to ensure compliance and availability of external data.

[0060] (2) When using external data, the data should be reviewed in accordance with the principles of "no access without demand" and "no access beyond the scope". After the review is passed, unified access can be made. During the review, in addition to the main business function fields, attention should also be paid to whether the reconciliation function is complete and whether the statistical reports are correct. Before data exchange, a data sharing and authorization agreement should be signed with the third party (i.e., the external data provider) to clarify the responsibilities and obligations of both parties for data security protection and ensure the security of data storage and application. Data use should be service-oriented, with unified access, unified storage and on-demand sharing, and security specifications can be managed.

[0061] (3) When evaluating external data, formulate relevant evaluation templates and evaluate external data in terms of data compliance, data quality, data services, and usage effects in combination with its usage scenarios. Based on the comprehensive evaluation results, timely perform external data withdrawal work to promote the effective update and iteration of data assets and improve the quality of data assets. Data evaluation at least includes data quality monitoring, service evaluation, application effect evaluation, provider evaluation, etc. Through regular audit summaries, the efficiency of unified management of external data can be improved.

[0062] The above-mentioned data access and integration module provided by the embodiment of the present invention, (1) through the use of low-code ideas, abstracts the real-time automatic analysis, conversion, cleaning, storage, association and other actions of external data into general components, reduces manual intervention and code development, and improves the efficiency of development and launch; realizes simulation interface testing, and uses baffle data to solve the problem of insufficient data in the test environment of some external data sources. (2) Monitors the status of external data services throughout the entire process, runs data verification rules in real time to monitor abnormal data, and ensures the stability of data services and the reliability of data. (3) Provides a complete audit mechanism, strictly follows the principle of "authorization first, then use", and ensures the controllability of data access. (4) Realizes external data lifecycle management, standardized data services, API services, etc.; integrates the billing engine to realize the calculation of external and internal expenses and the statistical analysis of internal cost sharing.

[0063] In one embodiment, the data storage module includes: an external data storage layer for caching external data using a Redis cluster.

[0064] In specific implementation, the external data storage layer can uniformly store external data and provide caching functions. The external data storage layer can use Redis cluster to cache external data and provide external data management layer queries on demand. Specifically, it can be set according to the frequency of queries and the size of the returned message. For example, the method for high-frequency queries is full storage, the method for high-frequency queries of large messages is valid field storage, and the method for low-frequency queries of large messages is index relationship storage. The principle of caching is that most data from external data sources have a validity period. Within the validity period, the same external data request will return the same result, so the results returned by the external data can be stored in the Redis cache according to different validity periods. When the same request is received within the validity period, the cached data can be directly obtained, saving costs while improving query efficiency.

[0065] Through unified data storage, the automatic local storage of real-time streaming data and offline file data in the external data service process simplifies the difficulty of complex processing such as data format conversion and data processing. By pre-setting multiple data sources and flexibly expanding computing components, the automation capability of external data integration and storage is improved.

[0066] The external data processing layer is used to perform structured parsing on the received external data messages, store the parsed external data in the external data storage layer, and perform standardized conversion on the external data message fields.

[0067] In specific implementation, the external data processing layer can realize the structuring and standardization of external data. The external data processing layer performs structured analysis on the received external data message and saves the parsed results in the external data storage layer. At the same time, the external data message fields are standardized and converted as needed. For example, different external data interface messages may have different names for ID card fields. They can be standardized and converted according to preset naming and code value specifications to unify internal and external data and facilitate rapid switching of external data sources. In addition, lightweight processing and cleaning can also be performed during message transmission, such as marking and completion of null values.

[0068] Specifically, during the data processing process, metadata can be used to write data processing scripts, clean, convert and integrate data. Building data according to business needs and using orchestrated source data for data processing mainly includes:

[0069] (1) Through the fully drag-and-drop data processing flow, users only need to drag the data processing components and configure the parameters to complete the ETL job development, with "zero" coding, easy to operate, easy to read, and easy to maintain.

[0070] (2) Supports process-based management of job trial runs, can conduct time-based or event-based debugging, can view execution logs and component description documents, and can preview post-run data online.

[0071] (3) It has built-in rich data processing and quality inspection analysis components, supporting various components such as SQL scripts, file storage, table migration, library migration, duplicate data check, logic check, linear regression, classification algorithm, etc.

[0072] (4) It can manually upload and automatically capture unstructured data files, automatically collect metadata, and automatically tag data. It can also preview .dat\pdf\doc and other files online.

[0073] (5) Component classification management and component customization can be performed. Based on the registration and definition of data processing components such as Spark and DataX, component types, component labels, data sources, user parameters, input controls, etc. can be defined. Users can flexibly adapt data processing components as needed.

[0074] (6) It is possible to define label templates, import labels in batches, and automatically generate labels. It is also possible to define attribute labels and basic label management, and manually or automatically label various data assets in batches.

[0075] (7) Build a data map and classify interfaces in multiple dimensions, including product dimension and manufacturer dimension, to help managers and users better understand the interfaces. In addition, the interfaces can be set with alternative options, so that backup plans can be set when accessing.

[0076] The external data presentation layer is used to view the basic information and called information of external data, and to verify the external data according to pre-set verification rules.

[0077] In specific implementation, the external data presentation layer can improve the monitoring and operation system. The external data presentation layer can provide management functions. For example, you can view the data source name, data provider, data type, number of interfaces, average response time, average response rate of the day, and the call status of each downstream business. In response to the needs of external data verification, you can set verification rules for the data returned by the external data source, including non-empty judgment, length setting, code value judgment, and custom rules, and perform data quality inspection and report export according to the set rules.

[0078] Furthermore, it also includes an external data management layer, which can separate specific business from original external data. The external data management layer includes: service timeout judgment, flow control, total control, service routing, access and permission control, automatic generation of interface code and other functions. Through configuration, interface code can be quickly generated to achieve rapid access to external data sources. At the same time, it supports visual orchestration services, and the system can intelligently select routes. For example, it can be set to sort by the cost of the data source. When calling the same data source, the data source with the lowest fee is given priority. It can also automatically determine the services that need to be throttled and limited, and configure the timeout time of various services according to the "channel + service" dimension.

[0079] In one embodiment, the data query and analysis module includes: a unified data service interface for different data sources, which is used for querying and retrieving external data. Specifically, the data query and analysis module has a unified standard data service interface, which is convenient for internal departments to access and use external data, and supports multiple data service methods, such as API interface, data visualization tools, etc., so that users can intuitively view and analyze data. In the embodiment of the present invention, flexible interface-level query is supported, including web page query interface; statistical analysis query, support interface call query according to different statistical calibers, time, product and other dimensions, as well as centralized management of providers and business needs; it can also provide a unified query interface for multiple data source service providers, support parameterized configuration of primary and standby interfaces, priority interfaces, and interface combination queries, and can adapt to various business scenarios. And by formulating a unified batch query architecture, it supports large-scale query services for business, can quickly respond to business needs, and broaden business application scenarios. In addition, it can also quickly screen and locate external data according to dimensions such as data source, data subject, data type, data layering, and data grouping, support full-text retrieval, and conduct joint query on metadata of data assets.

[0080] The call service management unit is used to provide call services to external data sources, as well as to manage the caller service interface, call details, call billing, statistical analysis, and batch call management. Specifically, the call service management unit provides caller service interface management, call details, call billing, statistical analysis, batch call, etc. It can also provide interface call capabilities to external platforms in the form of services, support interface serial and parallel configuration, interface shunting, intelligent routing switching strategy configuration, and support automatic generation and download of interface documents.

[0081] The derived indicator management unit is used for message structure management, offline message management, indicator template management, derived indicator management, derived indicator calculation and monitoring analysis. Specifically, the derived indicator management unit provides the functions of message call, processing and derived indicator processing, mainly including: message structure management, offline message management, indicator template management, derived indicator management, derived indicator calculation and monitoring analysis and other functions.

[0082] The data analysis unit includes a variety of data analysis tools and data analysis models, which are used to perform data analysis based on the user's data needs. Specifically, the data analysis unit discovers knowledge from the data based on the use of data analysis tools such as statistics, pattern recognition, machine learning, and data abstraction, directly or indirectly improving work efficiency, thereby assisting users to obtain correct data information and quickly make correct decisions. At the same time, the system also supports SQL and Python extensibility, provides statistical analysis and feature engineering, supports text analysis (such as word segmentation, word bag, word frequency, etc.), algorithms (regression, clustering, association, prediction algorithms), and automatic algorithm adjustment to participate in the self-learning of the result model. In addition, the preset data analysis model can quickly output visual analysis indicator data to form a visual report, providing authoritative and visual external data support for business operations and management decisions. Based on the statistical analysis of internal and external orders and bills, it can realize the productization of external data and quickly respond to business needs.

[0083] Furthermore, the data query and analysis module can also provide data asset cataloging from different perspectives. Users can create asset cataloging on demand to meet the data management needs of different roles. It supports adding new cataloging, setting attribution paths, and dynamic or static cataloging of each level of cataloging. After the cataloging is approved and released, the cataloging API can be automatically generated to facilitate the viewing and use of data asset cataloging.

[0084] The data query and analysis module uses a visual report dashboard to display reports such as interface call volume, distribution, and call exception information from different dimensions such as the caller and provider of the data interface service. Through the visual dashboard, the external data service situation is comprehensively managed from multiple perspectives. The diversified and professional intuitive visual charts can meet the real-time monitoring of data. At the same time, it can also perform visual analysis on the core indicators of platform operation such as server resource monitoring, application service link monitoring, and access link analysis. The data query and analysis module has built-in report templates for indicators such as provider orders, service bills, application visits, rankings, billing statistics, procurement management, and data flow. Managers can learn about the effects of external data services; at the same time, it provides operation and maintenance visualization capabilities such as server resource health status monitoring, service call link tracking, and performance analysis, so as to continuously improve external data business analysis and operational decision-making capabilities.

[0085] The data query and analysis module has built-in data quality assessment indicator rules, which can flexibly configure the rule types of quality assessment and automatically output assessment reports to objectively evaluate the quality of external data, reducing the difficulty of data quality assessment.

[0086] In one embodiment, see Figure 2As shown, the above system also includes: a data security and authority management module 40, which is used to adopt data encryption algorithms and security protocols to transmit, store and process data, and configure user data access rights.

[0087] In specific implementation, the data security and authority management module 40 uses encryption technology and security protocols to encrypt sensitive data according to unified data encryption rules and algorithms to prevent data from being intercepted and obtained during storage and transmission, and to ensure the data security of external data in storage, transmission and other links; through authority management and access control mechanisms, it ensures that only authorized users can access specific data resources. It can also record user operation logs and audit information to track and monitor data usage.

[0088] Specifically, building a monitoring and alarm mechanism includes the following specific processes:

[0089] (1) Data asset analysis and monitoring: Asset monitoring in multiple dimensions is performed through charts, and comprehensive analysis and monitoring of asset data volume, asset distribution, asset type ratio, asset and metadata usage, etc. is performed to facilitate users to clearly and comprehensively manage data assets.

[0090] (2) Authority control management: Utilize the data domain and data domain role definition management function to control various data domain management permissions through the attribution path, meet the multi-level and multi-dimensional authority management from data domain to role and user level, and different roles see different functional modules. Authorize users to view the specified data and tasks, and add, delete, modify and other operations on data and tasks according to the different authority granularity to meet security and confidentiality requirements.

[0091] (3) Data security control: Centrally manage, store and review data security management specification documents. Automatically discover and classify sensitive data and permissions, identify compliance risks, monitor sensitive data, monitor improper access to data, understand who is accessing the data, observe anomalies and prevent data loss. Use automated advanced analysis and machine learning to identify and stop abnormal high-risk behaviors. At the same time, the platform supports encryption of the platform's own data assets and flexibly sets up desensitization for platform data to ensure that the desensitization effect can meet the needs of various scenarios.

[0092] (4) Data service operations: Supports query by keyword and combined condition query, and can quickly query and retrieve target data.

[0093] (5) Permission management: Access the financial institution’s employee center to set, manage, allocate, and set permissions for roles; flexibly control interface call permissions by system, department, and other dimensions.

[0094] In one embodiment, see Figure 2As shown, the system further includes: a data export and sharing module 50, which is used to export external data according to a preset template. In specific implementation, when a user needs to export external data to other systems or share it with other users, the data export and sharing module 50 can import an EXCEL file according to a preset template, select the format, range and destination of the exported data, and set corresponding export options and parameters, and then convert and export the data in an appropriate format, and ensure the consistency and integrity of the exported data.

[0095] In one embodiment, see Figure 2 As shown, the above system also includes: a system maintenance and optimization module 60, which is used to monitor the system operation status and perform system operation and management based on the system operation status. In specific implementation, the system maintenance and optimization module 60 can regularly monitor the system operation status and performance, and promptly discover and solve potential problems and failures; it can also regularly upgrade and optimize the system to adapt to the ever-changing business needs and technical environment. Through regular maintenance and optimization, the system can maintain efficient and stable operation, providing users with a better data management experience.

[0096] Specifically, the system maintenance and optimization module 60 provides system operation and maintenance tools and methods to facilitate operation and maintenance personnel to perform system maintenance and management, mainly including:

[0097] (1) Monitoring and early warning: business usage monitoring, early warning of cumulative query volume and cumulative query fee exceeding, real-time interface service call exception monitoring, heartbeat monitoring, etc., used to monitor the operation of the system and the load of the server.

[0098] Specifically, through the pre-configured threshold of the total number of calls for each interface, a graphical display of the total number of calls for different interfaces is performed, and corresponding alarms or fuses are issued for those exceeding the corresponding level thresholds. Threshold configuration is performed for the call volume of different interfaces, such as setting a threshold of 80% as normal, an alarm is issued when it exceeds 80%, and a fuse is issued when it reaches 100%; alarm thresholds are set for CPU, disk space, memory, etc., and the resources of each server are monitored and graphically displayed, with normal and alarms displayed in different colors.

[0099] (2) Log query: Query system logs and business logs through the web page, mainly for the records of various interaction logs and system operation status records. Log the operation records, log each service, and monitor each transaction in the entire chain.

[0100] (3) Process approval: interface access, budget application and other approval processes.

[0101] The external data management system provided by the embodiment of the present invention has the following beneficial effects:

[0102] (1) Ability to quickly meet user needs: Built-in commonly used third-party data source interfaces, 95% of business indicators can be generated with zero code, support integrated interface orchestration, reduce the complexity of business system docking interfaces, data interfaces and indicator configurations can be exported on demand, realize cross-environment import and reuse, reduce configuration complexity, complete rapid launch, realize indicator cold start, meet the needs of rapid use of common application scenarios, save launch time, and realize the generation of indicators and data source interfaces through templated configuration, reduce the usage threshold of business personnel and the complexity of business system docking interfaces, and meet the needs of continuous access to data source interfaces.

[0103] (2) Improved data quality: By introducing data cleaning and standardization algorithms and setting flexible configuration quality assessment rules, the system can automatically identify and correct errors and inconsistencies in the data, achieving the effect of automatically detecting the quality of external data and the quality of data services provided by various external service providers, thereby improving the accuracy and reliability of the data. This helps companies make more informed decisions and reduce errors and losses caused by data quality issues.

[0104] (3) Simplified system integration costs: The system provides a variety of interfaces and flexible data format conversion functions, and provides data analysis and consulting services based on business data, the People's Bank of China's credit information, and other third-party external data, making integration with other systems simpler and more efficient. This reduces the technical difficulty and cost of integration and accelerates the digital transformation process of enterprises.

[0105] (4) Improved data processing and analysis capabilities: Based on the data requirements of business scenarios, by adopting a high-performance data processing and analysis engine, the system provides stability, timeliness, and coverage during the use of third-party data and indicators, and monitors and analyzes trends and abnormal values. The system can support rapid processing and complex analysis of large-scale data, which helps enterprises to deeply explore the value of data.

[0106] (5) Optimized costs and user experience: By optimizing the system architecture and algorithms, supporting conditional interface orchestration and diversion mechanisms, and implementing on-demand data source invocation, the system reduces operation and maintenance costs, thus saving costs for the enterprise. At the same time, the concise and clear user interface and operation process reduce the user's learning cost and difficulty of use, thus improving the user experience and satisfaction.

[0107] For the external data management system provided in the above embodiment, the present invention also provides an external data management method, see Figure 3 The flowchart of an external data management method shown in FIG. 1 illustrates that the method mainly includes the following steps S301 to S303:

[0108] Step S301: acquiring external data from different external data sources using a preset collection standard through a data access and integration module, and cleaning and integrating the external data;

[0109] Step S302: storing the integrated external data through the data storage module, and performing structured and standardized processing on the external data;

[0110] Step S303: Retrieve, query and analyze the external data of the data storage module based on the user's data requirements through the data query and analysis module.

[0111] The above-mentioned external data management method provided by the present invention establishes unified external data collection standards and specifications through a data access and integration module, obtains external data from different external data sources, integrates the external data, and then structures and standardizes the obtained external data through a data storage module; finally, the external data of the data storage module can be retrieved, queried and analyzed according to the user's data requirements, thereby realizing unified management of external data, ensuring the integrity and consistency of the data, and improving the efficiency of using external data.

[0112] It should be noted that the implementation principle and technical effects of the method provided in the embodiment of the present invention are the same as those of the aforementioned system embodiment. For the sake of brief description, for matters not mentioned in the method embodiment, reference may be made to the corresponding contents in the aforementioned system embodiment.

[0113] An embodiment of the present invention further provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above implementation methods.

[0114] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 41, a memory 42, a bus 43 and a communication interface 44, wherein the processor 41, the communication interface 44 and the memory 42 are connected via the bus 43; the processor 41 is used to execute an executable module stored in the memory 42, such as a computer program.

[0115] The memory 42 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 44 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0116] The bus 43 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0117] Among them, the memory 42 is used to store programs, and the processor 41 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 41, or implemented by the processor 41.

[0118] The processor 41 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 41. The above processor 41 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 42, and the processor 41 reads the information in the memory 42 and completes the steps of the above method in combination with its hardware.

[0119] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0120] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0121] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. An external data management system, characterized in that: include: Data access and integration module, data storage module and data query and analysis module; The data access and integration module uses preset collection standards to obtain external data from different external data sources, and cleans and integrates the external data; The data storage module stores the integrated external data and performs structured and standardized processing on the external data; The data query and analysis module retrieves, queries and analyzes the external data of the data storage module based on the user's data requirements.

2. The system according to claim 1, characterized in that The data access and integration module includes: a data source management unit, which is used for external data source management and establishing an external data lifecycle management system; wherein, the external data lifecycle management system includes: when external data is introduced, the provider of the external data is approved according to the standardized approval process and document template, and the external data is introduced into the process for merging according to data requirements; when external data is used, the user's data requirements are reviewed according to pre-set review principles, and external data is uniformly obtained from the data storage module after the review is passed; when external data is evaluated, the external data is evaluated according to a pre-set evaluation template; wherein data evaluation at least includes data quality monitoring, service evaluation, application effect evaluation, and provider evaluation.

3. The system according to claim 1, characterized in that The data storage module comprises: The external data storage layer is used to cache external data using the Redis cluster; The external data processing layer is used to perform structural analysis on the received external data message, store the analyzed external data in the external data storage layer, and perform standardized conversion on the external data message fields; The external data presentation layer is used to view the basic information and called information of the external data, and to verify the external data according to pre-set verification rules.

4. The system according to claim 1, characterized in that The data query and analysis module includes: a unified data service interface for different data sources, used for querying and retrieving external data; The call service management unit is used to provide call services to external data sources, and to manage the caller service interface, call details, call billing, statistics and analysis, and batch call management; Derived indicator management unit, used for message structure management, offline message management, indicator template management, derived indicator management, derived indicator calculation and monitoring analysis; The data analysis unit includes a variety of data analysis tools and data analysis models, which are used to perform data analysis based on the user's data requirements.

5. The system according to claim 1, characterized in that Also includes: The data security and rights management module is used to adopt data encryption algorithms and security protocols for data transmission, storage and processing, as well as to configure user data access rights.

6. The system according to claim 1, characterized in that Also includes: The data export and sharing module is used to export the external data according to a preset template.

7. The system according to claim 1, characterized in that Also includes: The system maintenance and optimization module is used to monitor the system operation status and perform system operation, maintenance and management based on the system operation status.

8. An external data management method, characterized in that: The external data management system as claimed in any one of claims 1 to 7 comprises: Acquire external data from different external data sources using the data access and integration module using preset collection standards, and clean and integrate the external data; The integrated external data is stored in the data storage module, and the external data is structured and standardized; The data query and analysis module is used to retrieve, query and analyze the external data of the data storage module based on the data requirements of the user.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the steps of the method according to claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 8 are performed.

Images