An information integration data processing method, system, electronic device, and storage medium

Abstract

This invention discloses an information integration data processing method, system, electronic device, and storage medium. The method includes: storing data resources from at least one business information system into a data resource pool; generating a data directory for the data resource pool according to a preset data model; determining a data access interface based on the data directory and a data access request; and accessing target data resources within the data resource pool based on the data access interface. This invention enables the integration of data resources from different business information systems, sharing these resources, facilitating data resource mining, improving data resource utilization efficiency, and enhancing the scientific management of business system information.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an information integration data processing method, system, electronic device, and storage medium. Background Technology

[0002] Information systems are a crucial foundation for the development and utilization of business information resources. By organizing, utilizing, and managing data resources through information systems, the integration of various elements can be achieved. Data resources, as the basis of information system construction, require integration and integrated application; therefore, data resource integration is increasingly becoming a key determining factor in informatization. Currently, the integration and processing of data resources within various information systems typically suffers from the following shortcomings:

[0003] 1. Multiple information systems are built independently and are not interconnected with external systems, resulting in incompatibility among most of them. The lack of a unified platform standard among the information systems makes it impossible for them to share data, which greatly affects the standardization and scientification of information management and seriously restricts the coordinated development of information technology construction.

[0004] 2. The sharing of data resources in information systems is not high. Data resources are stored within each information system, and there is a lack of effective mining and utilization of data resources, resulting in low utilization rate of data resources and a lack of scientific data resource management within information systems. Summary of the Invention

[0005] This invention provides an information integration data processing method, system, electronic device, and storage medium to achieve data resource integration between different information systems, realize data resource sharing between different systems, facilitate data resource mining, improve the utilization efficiency of data resources, and enhance the scientific management of information.

[0006] According to one aspect of the present invention, an information integration data processing method is provided, wherein the method includes:

[0007] Store the data resources of at least one business information system in a data resource pool;

[0008] Generate the data catalog of the data resource pool according to the preset data model;

[0009] The data access interface is determined based on the data catalog and the data access request;

[0010] Access the target data resources within the data resource pool based on the data access interface.

[0011] According to another aspect of the present invention, an information integration data processing system is provided, wherein the system comprises:

[0012] A data storage module is used to store data resources from at least one business information system into a data resource pool.

[0013] The directory generation module is used to generate the data directory of the data resource pool according to a preset data model;

[0014] An interface determination module is used to determine the data access interface based on the data directory and the data access request.

[0015] The resource access module is used to access target data resources within the data resource pool based on the data access interface.

[0016] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the information integration data processing method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the information integration data processing method according to any embodiment of the present invention.

[0021] The technical solution of this invention stores data resources from different business information systems in a data resource pool, generates a data catalog for the data resource pool according to a preset data model, determines the data access interface according to the data target and data access request, and realizes access to the target data resources in the data resource pool by calling the data access interface. This achieves data resource integration of different business information systems, realizes data resource sharing between different systems, helps to realize data resource mining, improves the utilization efficiency of data resources, and enhances the scientific management of system information.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of an information integration data processing method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of an information integration data processing method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of an information integration data processing system according to Embodiment 3 of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of an electronic device that implements the information integration data processing method of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1This is a flowchart of an information integration data processing method according to Embodiment 1 of the present invention. This embodiment is applicable to the sharing and utilization of data resources within different business information systems. The method can be executed by an information integration data processing system, which can be implemented in hardware and / or software and can be configured within a server or server cluster. Figure 1 As shown, the method includes:

[0032] Step 110: Store the data resources of at least one business information system into the data resource pool.

[0033] The business information system can be an information system for various professional fields. The development methods and hardware / software architectures of these systems can differ, and they can belong to different organizations. In some embodiments, the business information system may include traditional relational database systems, file systems, network message systems, distributed messaging systems, distributed databases, distributed file systems, etc. It is understood that the development methods and hardware / software architectures of these business information systems can all be different. Data resources can be valuable data generated during the operation of the business information system. Data resources can include automated and non-automated data, and can include different data types according to the different functions implemented by the business information system. Data resources can include structured and unstructured data.

[0034] In this embodiment of the invention, communication connections can be established with different business information systems. Data resources from these systems can be acquired within these connections and stored in a data resource pool. The data resource pool can be a shared collection of data resources, specifically a data warehouse or data lake. It can be built using a distributed architecture, connecting different business information systems and eliminating data silos. The data resource pool supports structured, semi-structured, and unstructured data. In some embodiments, the data resource pool can be built using a Hadoop distributed cluster, allowing data resources from various business information systems to be organized and stored within the pool.

[0035] Step 120: Generate the data catalog of the data resource pool according to the preset data model.

[0036] The preset data model can be generated by abstracting data characteristics based on professional domain information. It can include a set of data concepts and definitions, and can consist of data structures, data operations, and data constraints. It is understood that there can be one or more preset data models, which can be abstracted and generated separately according to the data characteristics of different data within different business information systems. The data directory is information used to manage the data resource pool. It can include one or more directory entries, each corresponding to a type of data resource. The data directory can be used for the management, analysis, and navigation of data resources. There can be one or more data directories, which can be divided into local directories, global directories, and conditionally shared directories according to their storage location.

[0037] In this embodiment of the invention, one or more pre-configured preset data models can be read. These preset data models can be generated by the administrator based on the data characteristics of different business information systems. Data resources within the data resource pool can be organized according to these preset data models, and the data resources can be stored as directory entries in a data directory according to different preset data models. It is understood that the data directory can be updated periodically or quantitatively after generation. In one embodiment, when new data resources are stored in the data resource pool, the directory entry of that data resource can be registered in the data directory. In another embodiment, the preset data models can be defined according to business affiliation. Data resources corresponding to different data assets can be queried within the data resource pool according to the preset data models, and these data assets can be organized into a data directory.

[0038] Step 130: Determine the data access interface based on the data catalog and data access request.

[0039] Data access requests can be user requests to access data resources. These requests can originate from different business information systems and can be generated according to the access format specified by the data resource pool or the access format of the business information system. In some embodiments, the data access requests obtained by the data resource pool can be formatted uniformly to ensure compatibility with the communication methods of different business information systems. Data access interfaces are function interfaces for accessing data within the data resource pool. These interfaces can be configured with data resource access rules and can be set according to the types of data resources within the data resource pool. It is understood that different data resources may have different data access interfaces.

[0040] In this embodiment of the invention, a data access request can be monitored. Upon receiving a data access request, the system can query the data resource to be accessed in the data directory according to the data access request, and obtain a matching data access interface based on the data resource. It is understood that there can be one or more data access requests, and the data resources corresponding to each data access request can be queried simultaneously in the data directory to improve data access efficiency. In some embodiments, the data directory can store the resource names of the data resources. The resource names in the data access request can be matched with the data directory to retrieve the data resource to be accessed. In some embodiments, if the data resource to be accessed is not found in the data directory, a data query error request can be sent back to the data source of the data access request.

[0041] Step 140: Access the target data resource in the data resource pool based on the data access interface.

[0042] Among them, the target data resource can be the data resource to be queried through a data access request. The data access request can be different from the data source of the target data resource. For example, the data access request comes from the first business information system, while the target data resource comes from the second business information system. The data resource pool can realize data sharing between different business information systems.

[0043] In this invention, data access interfaces can be invoked to access target data resources in the data resource pool through the data access rules configured within the data access interfaces. It is understood that upon reading the target data resource, the target data can be processed according to the content of the data access request, including but not limited to data updates, data deletions, and data additions. In some embodiments, different data access interfaces are configured with data resource access rules for different data structures, allowing access to data resources with different data structures by invoking the data access interfaces. In other embodiments, data access interfaces are configured with data resource access rules for different file systems, and the data resource pool consists of different data storage systems. Access to traditional relational database data, file system data, network message system data, distributed message queue data, distributed database data, and distributed file system data can be achieved by invoking the data access interfaces, thereby realizing integrated management of data resources and improving compatibility between different business information systems.

[0044] In this embodiment of the invention, data resources from different business information systems are stored in a data resource pool. A data catalog is generated for the data resource pool according to a preset data model. Data access interfaces are determined according to data targets and data access requests. Access to target data resources within the data resource pool is achieved by calling the data access interfaces. This enables the integration of data resources from different business information systems, the sharing of data resources between different systems, and facilitates data resource mining. It can also improve the utilization efficiency of data resources and enhance the scientific management of system information.

[0045] Example 2

[0046] Figure 2 This is a flowchart of an information integration data processing method according to Embodiment 2 of the present invention. This embodiment is a specific modification based on the above-described embodiments. See also... Figure 2 The specific steps in this embodiment of the invention are as follows:

[0047] Step 210: Read the data configuration information of the business information system.

[0048] The data configuration information can be pre-set parameter information for accessing different business information systems. The data configuration information can include the permission authentication information and communication protocols of different business information systems. The data configuration information can exist in the form of an encrypted file.

[0049] In this embodiment of the invention, data configuration information from different business information systems can be read. This data configuration information can be located in the same or different storage locations. In some embodiments, data configuration files from different business information systems can be stored in the same configuration file, allowing different data configuration information to be read from a single configuration file. In other embodiments, the data configuration information can be stored in the hardware keys of different users, allowing the data configuration information of that business information system to be read after the hardware key is accessed.

[0050] Step 220: Determine the data access channel according to the data configuration information in the preset configuration file.

[0051] The preset configuration file can be a data file containing access rules for different business information systems. This preset configuration file can be configured within the server or server cluster executing the method of this invention. The preset configuration file includes one or more configuration items, each corresponding to an access rule for a business information system. It is understood that this access rule may include a data communication protocol. The data access channel can be a data transmission channel established with the business information system. Specifically, the data access channel may include nested text data transmission channels, hypertext transfer protocol channels, streaming media transmission protocol channels, etc.

[0052] In this embodiment of the invention, a preset configuration file can be loaded into memory. One or more data access channel construction rules can be found within the preset configuration file according to the read data configuration information. A data access channel can be constructed according to these construction rules and the data configuration information, along with the business information system. It is understood that determining the data access channel in the preset configuration file based on the data configuration information may include finding the identifier of the business information system corresponding to the data configuration information and finding the data access channel associated with that identifier. In some embodiments, the data access channel has been pre-established, and access permissions for the data access channel can be obtained according to the acquired data configuration information.

[0053] Step 230: Read the data resources of the business information system within the data access channel and store the data resources in the data resource pool.

[0054] In this embodiment of the invention, data resources can be obtained from the business information system through a data access channel. It is understood that the data resources can be actively pushed by the business information system through the data access channel or retrieved by the execution device of the method of this invention from the business information system through the data access channel. After obtaining the data resources, they can be stored in a data resource pool. In some embodiments, access permissions for data resources can be set within the business information system to control data resource sharing. Only data resources with shared permissions set within the business information system can be obtained through the data access channel, thereby improving data resource security while ensuring data sharing and utilization.

[0055] Step 240: Read the data model definition of the preset data model, wherein the data model definition includes at least the data structure.

[0056] The data model definition can include definitional information describing different data resources. It consists at least of a data structure, which describes the static characteristics of the data resources, including their type, content, properties, and relationships. These relationships can include hierarchical and relational structures between different data resources. The data model definition can also include information on data operations and constraints. Data operations describe the dynamic characteristics of the data resources, including data insertion, modification, deletion, and querying. Data constraints can include integrity rules for the data resources, which are constraints and storage rules governing the data resources and their relationships; for example, the data resource's value cannot be negative or its data type is integer.

[0057] In this embodiment of the invention, a corresponding data model definition can be extracted for each preset data model. This data model definition can be abstracted and generated by the administrator based on the data characteristics of the business information system. Each data model definition can consist of elements such as data structure, data operations, and data constraints. It is understood that the data model definition can exist in the form of a configuration file.

[0058] Step 250: Locate the data resources corresponding to each data model definition in the data resource pool.

[0059] Specifically, data resources can be categorized within the data resource pool according to data model definitions. Each data model definition can have one or more corresponding data resources. Data resources within each category can all satisfy the element information defined by the corresponding data model, including but not limited to data structure, data operations, and data constraints. In one embodiment, the data structure of each data model definition can be extracted, and the corresponding data resources can be queried within the data resource pool according to the extracted data structure.

[0060] Step 260: For each data resource, determine at least one hierarchical relationship based on the data structure.

[0061] Among them, the association hierarchy can be information that reflects the interrelationship between different data resources, and the association hierarchy can reflect whether there is an association between at least two data resources.

[0062] In this embodiment of the invention, for data resources belonging to different data model definitions, the data structure within the data model definition of that type of data resource can be extracted, and the association hierarchy with at least one other data resource can be determined according to the static characteristics within that data structure. For example, when a data resource can be associated with data resource A according to its data structure, identification information can be created for that data resource and data resource A to represent the association hierarchy.

[0063] Step 270: Generate the data catalog corresponding to each data resource according to the relationship of each association level.

[0064] In this embodiment of the invention, the hierarchical architecture between different data resources can be determined according to the determined association hierarchy, and the data resources can be converted into directory items according to the hierarchical architecture, and the directory items can be stored as a data directory according to the hierarchical architecture.

[0065] Step 280: Extract the data feature identifier from the data access request.

[0066] Among them, data feature identifiers can be information that reflects the characteristics of data resources, and may include information such as data name, data type, and business affiliation.

[0067] In this embodiment of the invention, data feature identifiers such as data name, data type, and business affiliation can be extracted from each data access request for matching data resources.

[0068] Step 290: Search for data resources that match the data feature identifier in the data catalog.

[0069] In this invention, data feature identifiers can be matched with directory entries within the data directory, and the data resources corresponding to the matched directory entries can be used as the data resources that match the data feature identifiers. It is understood that this matching process can be implemented by matching the directory entries with the data feature identifier strings or by using regular expressions.

[0070] Step 2100: Extract the data access interface that matches the data resource from the preset interface file.

[0071] In this embodiment of the invention, the preset interface file may be a configuration file that stores data access interfaces. The preset interface file may include one or more data access interfaces, and the matching data access interface can be found according to the resource name or resource type of the data resource.

[0072] Step 2110: Access the target data resource in the data resource pool based on the data access interface.

[0073] This invention, in its embodiments, reads data configuration information from different business information systems, determines access channels within a preset configuration file according to the data configuration information, acquires data resources from the business information systems through the data access channels, and stores the data resources in a data resource pool. It then reads data model definitions containing data structures, searches the data resource pool for data resources that match each data model definition, generates a hierarchical relationship for each data resource according to the data structure within the corresponding data model definition, processes the data resources into a data directory according to this hierarchical relationship, extracts data feature identifiers from data access requests, searches the data directory for data resources matching the data feature identifiers, acquires a data access interface matching the data resources, and accesses the target data resources in the data resource pool by calling the access interface. This achieves data resource integration and sharing among different business information systems, facilitates data resource mining, improves data resource utilization efficiency, and enhances the scientific management of system information.

[0074] In some embodiments, before storing the data resources into the data resource pool, the method further includes:

[0075] Extract the data structure of the data resource and determine the preset format conversion rules according to the data structure; convert the data format of the data resource according to the preset conversion rules.

[0076] Among them, the preset format conversion rules can be rules for converting the format of data resources. The preset format conversion rules can include the source data format and the target data format.

[0077] In this embodiment of the invention, the data format of the data resources stored in the data resource pool can be unified, and corresponding preset conversion rules can be extracted for data resources with different data structures. The data resources can be converted from the source data format to the target data format according to the preset conversion rules.

[0078] In other embodiments, generating the data directory corresponding to each of the data resources according to the respective association hierarchy includes:

[0079] Generate a minimum connected graph from the data resources and the associated hierarchical relationships; and use each of the data resources as a directory entry in the data directory according to the minimum connected graph.

[0080] Among them, the minimum connected graph can be a graph composed of data resources as nodes and management hierarchy relationships as edges. This minimum connected graph can reflect the simplest association relationship between data resources.

[0081] In this embodiment of the invention, a minimum connected graph can be constructed using data resources as nodes and hierarchical relationships as edges. This construction process can be generated using Prim's algorithm or Kruskal's algorithm. The data resources of each node in the minimum connected graph can be used as directory items, and the directory items can be organized into a data directory according to the hierarchical structure of the minimum connected graph. It is understood that the hierarchical relationship of different directory items in the data directory is the same as the hierarchical relationship of the nodes of each directory item in the minimum connected graph.

[0082] In some embodiments, the method further includes: generating data quality analysis results of the data resources according to a preset data granularity, wherein the preset data granularity includes at least null values, duplicate values, format, reference values, and volatility.

[0083] In this embodiment of the invention, data quality analysis can be performed on each data resource. During the data quality analysis, preset data granularities such as null values, duplicate values, format, reference values, and volatility within the data resources can be analyzed and statistically analyzed. Different preset data granularities can have different weight values, and the sum of the weight values ​​of each data resource can be used as the data quality analysis result.

[0084] Example 3

[0085] Figure 3 This is a schematic diagram of the structure of an information integration data processing system provided according to Embodiment 3 of the present invention. Figure 3As shown, the system includes: a data storage module 310, a directory generation module 320, an interface determination module 330, and a resource access module 340.

[0086] Data storage module 310 is used to store data resources of at least one business information system into a data resource pool.

[0087] The catalog generation module 320 is used to generate the data catalog of the data resource pool according to the preset data model.

[0088] The interface determination module 330 is used to determine the data access interface based on the data directory and the data access request.

[0089] The resource access module 340 is used to access the target data resources in the data resource pool based on the data access interface.

[0090] In this embodiment of the invention, a data storage module stores data resources from different business information systems in a data resource pool. A directory generation module generates a data directory for the data resource pool based on a preset data model. An interface determination module determines the data access interface according to the data target and data access request. A resource access module accesses the target data resources in the data resource pool by calling the data access interface. This enables the integration of data resources from different business information systems, achieves data resource sharing between different systems, facilitates data resource mining, improves the utilization efficiency of data resources, and enhances the scientific management of system information.

[0091] Furthermore, based on the embodiments of the present invention, the data storage module 310 includes:

[0092] The information reading unit is used to read the data configuration information of the business information system.

[0093] The channel determination unit is used to determine the data access channel according to the data configuration information in the preset configuration file.

[0094] A resource storage unit is used to read the data resources of the business information system within the data access channel and store the data resources in the data resource pool.

[0095] Furthermore, based on the above embodiments of the invention, the system further includes: a format conversion module, used to extract the data structure of the data resource, determine a preset format conversion rule according to the data structure, and perform data format conversion on the data resource according to the preset conversion rule.

[0096] Furthermore, based on the above embodiments of the invention, the catalog generation module 320 includes:

[0097] Define a reading unit for reading the data model definition of the preset data model, wherein the data model definition includes at least a data structure.

[0098] The resource classification unit is used to locate the data resources corresponding to each of the data model definitions in the data resource pool.

[0099] A hierarchical relationship unit is used to determine at least one hierarchical relationship of the data resources according to the data structure for each of the data resources.

[0100] The directory generation unit is used to generate the data directory corresponding to each of the data resources according to the hierarchical relationship.

[0101] Furthermore, based on the above embodiments of the invention, the directory generation unit is specifically used to: generate a minimum connected graph from the data resources and the associated hierarchical relationship; and use each of the data resources as a directory entry of the data directory according to the minimum connected graph.

[0102] Furthermore, based on the above embodiments of the invention, the interface determination module 330 includes:

[0103] The identifier extraction unit is used to extract the data feature identifier within the data access request.

[0104] The directory lookup unit is used to search for the data resource that matches the data feature identifier within the data directory.

[0105] An interface extraction unit is used to extract the data access interface that matches the data resource from a preset interface file.

[0106] Furthermore, based on the above embodiments of the invention, the system further includes: a data quality module, used to generate data quality analysis results of the data resources according to a preset data granularity, wherein the preset data granularity includes at least null values, duplicate values, format, reference values, and volatility.

[0107] The information integration data processing system provided in the embodiments of the present invention can execute the information integration data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0108] Example 4

[0109] Figure 4This is a schematic diagram of the structure of an electronic device implementing the information integration data processing method of the embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0110] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0111] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as information integration data processing methods.

[0113] In some embodiments, the information integration data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the information integration data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the information integration data processing method by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0119] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0120] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. An information integration data processing method, characterized in that, include: Store the data resources of at least one business information system in a data resource pool; Generate the data catalog of the data resource pool according to the preset data model; The data access interface is determined based on the data catalog and the data access request; Access the target data resources in the data resource pool based on the data access interface; The step of generating the data catalog of the data resource pool according to the preset data model includes: Read the data model definition of the preset data model, wherein the data model definition includes at least a data structure; Search the data resource pool for the data resource corresponding to each of the data model definitions; For each of the data resources, at least one hierarchical relationship is determined according to the data structure; The data catalog corresponding to each data resource is generated according to the hierarchical relationship described above.

2. The method according to claim 1, characterized in that, The step of storing data resources from at least one business information system into a data resource pool includes: Read the data configuration information of the business information system; The data access channel is determined according to the data configuration information in the preset configuration file; The data resources of the business information system are read within the data access channel, and the data resources are stored in the data resource pool.

3. The method according to claim 2, characterized in that, Before storing the data resources into the data resource pool, the method further includes: Extract the data structure of the data resource and determine the preset format conversion rules according to the data structure; The data resources are converted according to the preset conversion rules.

4. The method according to claim 1, characterized in that, The step of generating the data directory corresponding to each data resource according to the hierarchical relationship includes: Generate a minimum connected graph from the data resources and the associated hierarchical relationships; Each of the data resources is used as a directory entry in the data directory according to the minimum connectivity graph.

5. The method according to claim 1, characterized in that, The step of determining the data access interface based on the data directory and the data access request includes: Extract the data feature identifier from the data access request; Search within the data catalog for the data resource that matches the data feature identifier; Extract the data access interface that matches the data resource from the preset interface file.

6. The method according to any one of claims 1-5, characterized in that, Also includes: Data quality analysis results of the data resources are generated according to a preset data granularity, wherein the preset data granularity includes at least null values, duplicate values, format, reference values, and volatility.

7. An information integration data processing system, characterized in that, include: A data storage module is used to store data resources from at least one business information system into a data resource pool. The directory generation module is used to generate the data directory of the data resource pool according to a preset data model; An interface determination module is used to determine the data access interface based on the data directory and the data access request. The resource access module is used to access target data resources within the data resource pool based on the data access interface. The step of generating the data catalog of the data resource pool according to the preset data model includes: Read the data model definition of the preset data model, wherein the data model definition includes at least a data structure; Search the data resource pool for the data resource corresponding to each of the data model definitions; For each of the data resources, at least one hierarchical relationship is determined according to the data structure; The data catalog corresponding to each data resource is generated according to the hierarchical relationship described above.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the information integration data processing method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the information integration data processing method according to any one of claims 1-6.

Images