Multi-source heterogeneous data processing method and device, electronic equipment and storage medium

Abstract

The invention provides a multi-source heterogeneous data processing method and device, electronic equipment and a storage medium, and is applied to a software operation system and a third-party system.The method comprises the steps that a transmission mode and a data format supported by data access of the software operation system are defined, and multi-source heterogeneous data are obtained from the third-party system according to the transmission mode and the data format; performing data processing on the multi-source heterogeneous data, wherein the data processing comprises data analysis, data cleaning, data conversion and data processing; performing hierarchical storage on the processed multi-source heterogeneous data according to data attributes; and obtaining post-monitoring configuration information of the multi-source heterogeneous data after data processing, and triggering a service logic operation according to the post-monitoring configuration information. According to the method, a network protocol is used for replacing a third-party system for data access, the reuse rate of codes is increased, the development workload is reduced, later iteration and extension of a software operation system are facilitated, and visualization, configuration and full-life-cycle management of data processing can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-source heterogeneous data processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of SaaS (Software as a Service) systems, it has become the norm to continuously connect to a large number of third-party systems and access their data for interactive operations. Due to differences in the interaction methods, communication mechanisms, data encryption and decryption algorithms, data volume, and data structure of different third-party systems, the complexity of data processing has increased, and the development workload of software operation systems has continued to grow, which is not conducive to the stability of existing systems.

[0003] At present, when processing a large number of data sources from different third-party systems, a processing method based on the adapter pattern is often used. The data from different data sources are converted, and each third-party system has an independent set of adapter processing strategies to convert the data into the data format required by the software operation system, so as to achieve the independence of data interaction between various third-party systems, and later changes will not affect each other. After data access, it is stored in the platform database, and the business system obtains its own data for business operations as needed. However, with the growth of business, each third-party system has independent strategies, which will lead to greater development and operation and maintenance difficulties, which is not conducive to the later iteration and expansion of the software operation system. Therefore, there is an urgent need for a data processing solution to solve the current problem. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a multi-source heterogeneous data processing method, device, electronic device and storage medium, which effectively solves the problem in the prior art that each third-party system has an independent strategy, resulting in a large amount of development and operation and maintenance difficulty, which is not conducive to the later iteration and expansion of the software operation system.

[0005] In a first aspect, the present invention provides a multi-source heterogeneous data processing method, which is applied to a software operation system and a third-party system, and comprises:

[0006] Define the transmission mode and data format supported by the data access of the software operation system, and obtain multi-source heterogeneous data from the third-party system according to the transmission mode and the data format;

[0007] The multi-source heterogeneous data is processed, including:

[0008] Perform data analysis according to the data conversion protocol and encryption and decryption rules configured in the third-party system;

[0009] Clean the data according to the data deduplication rules, necessary attributes and metadata definitions;

[0010] Convert the cleaned data according to the third-party system and business type;

[0011] Perform data processing on the data after data conversion;

[0012] The processed multi-source heterogeneous data is stored in layers according to data attributes;

[0013] Acquire post-monitoring configuration information of the multi-source heterogeneous data after the data processing, and trigger a business logic operation according to the post-monitoring configuration information.

[0014] Furthermore, the data parsing according to the data conversion protocol and encryption and decryption rules configured in the third-party system includes:

[0015] Acquiring configuration information of the third-party system according to each data source of the multi-source heterogeneous data;

[0016] Obtain an encryption algorithm instance object according to the configuration information;

[0017] The encrypted data in each of the data sources is decrypted according to the encryption algorithm instance object to obtain plaintext data.

[0018] Furthermore, the data cleaning is performed according to the data deduplication rules, necessary attributes and metadata definitions, including:

[0019] Deduplication of the plaintext data is performed according to a deduplication rule;

[0020] Filter the deduplicated plaintext data according to the necessary metadata attributes;

[0021] Format requirements are set for the filtered plaintext data according to the regular expression, and difference data that does not meet the format requirements are filtered out from the filtered plaintext data to obtain standard data.

[0022] Furthermore, the data after cleaning is converted according to the third-party system and business type, including:

[0023] Acquire a metadata conversion configuration table according to the third-party system;

[0024] The standard data is converted according to the metadata conversion configuration table to obtain target data.

[0025] Furthermore, the data processing after the data conversion includes at least one of the following steps:

[0026] Setting default values ​​for some attributes of the target data according to global default values;

[0027] Supplementing tenant information for the target data according to tenants of the third-party system and the software operation system;

[0028] A data label is established for the target data according to the data value.

[0029] Furthermore, storing the processed multi-source heterogeneous data in layers according to data attributes includes:

[0030] storing the configuration data and temporary data in the multi-source heterogeneous data after the data processing in the cache middleware;

[0031] Establishing a relational database, storing basic data and business data in the multi-source heterogeneous data after the data processing in the relational database;

[0032] A data cluster is constructed, and the full life cycle data of the multi-source heterogeneous data after the data processing is stored in the data cluster.

[0033] Furthermore, the transmission modes supported by the data access include at least one or more of the following modes: http protocol, transport layer protocol, intermediate library, message queue and text data.

[0034] In a second aspect, the present invention provides a multi-source heterogeneous data processing device, which is applied to a software operation system and a third-party system, and includes:

[0035] A data access module, used to define the transmission mode and data format supported by the data access of the software operation system, and to obtain multi-source heterogeneous data from the third-party system according to the transmission mode and the data format;

[0036] A data processing module is used to process the multi-source heterogeneous data, including:

[0037] Perform data analysis according to the data conversion protocol and encryption and decryption rules configured in the third-party system;

[0038] Clean the data according to the data deduplication rules, necessary attributes and metadata definitions;

[0039] Convert the cleaned data according to the third-party system and business type;

[0040] Perform data processing on the data after data conversion;

[0041] A data storage module, used for storing multi-source heterogeneous data after data processing in layers according to data attributes;

[0042] The post-trigger module is used to obtain the post-monitoring configuration information of the multi-source heterogeneous data after the data processing, and trigger the business logic operation according to the post-monitoring configuration information.

[0043] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multi-source heterogeneous data processing method as described in the first aspect of the present invention.

[0044] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the multi-source heterogeneous data processing method as described in the first aspect of the present invention is implemented.

[0045] The multi-source heterogeneous data processing method, device, electronic device and storage medium provided by the present invention use network protocols instead of third-party systems for data access, which greatly improves the reuse rate of codes, reduces development workload, and realizes basic functions while using redis, xml and message queues and other technical means to realize the processing of complex business, ensuring that the data is completely processed into data that meets business specifications. The attribute value replacement of entity objects is replaced by configuration files, and data conversion is realized with a higher abstract dimension. Data cleaning and data conversion can efficiently filter out effective data, improve data quality, and greatly improve the accuracy of conversion. Relying on the statelessness of the present invention, it can be infinitely expanded horizontally to achieve functions such as high performance, high availability and dynamic scaling. It has strong universality and configurability, and can realize visualization, configuration and life cycle management of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 It is a first schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0048] Figure 2 This is a second schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0049] Figure 3 It is a third schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0050] Figure 4It is a fourth schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0051] Figure 5 is a fifth schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0052] Figure 6 It is a sixth schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0053] Figure 7 This is the seventh schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0054] Figure 8 is a schematic diagram of the structure of a multi-source heterogeneous data processing device provided by an embodiment of the present invention;

[0055] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention.

[0056] Description of main component symbols:

[0057] 800, multi-source heterogeneous data processing device; 810, data access module; 820, data processing module; 830, data storage module; 840, post-trigger module; 900, electronic device; 910, processor; 920, communication interface; 930, memory; 940, communication bus. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be further clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. It should be noted that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more of the related listed items.

[0061] The SaaS system defines standard business process specifications and data specifications. The implementation and execution of the specifications depend on the consistency of the data. The data access gateway serves as the hub for data transmission between systems. Its purpose is to ensure that the data transmitted to the business system has the same structure and meets the access requirements and specifications. It also reduces the coupling between systems and is the basis for the stable operation of the system.

[0062] At present, when processing a large number of data sources from different third-party systems, a processing method based on the adapter pattern is often used. The data from different data sources are converted, and each third-party system has an independent set of adapter processing strategies to convert the data into the data format required by the software operation system, so as to achieve the independence of data interaction between various third-party systems, and later changes will not affect each other. After data access, it is stored in the platform database, and the business system obtains its own data for business operations as needed. However, with the growth of business, each third-party system has independent strategies, which will lead to greater development and operation and maintenance difficulties, which is not conducive to the later iteration and expansion of the software operation system. Therefore, there is an urgent need for a data processing solution to solve the current problem.

[0063] Example 1

[0064] The embodiment of the present invention provides a multi-source heterogeneous data processing method, which effectively solves the problem in the prior art that each third-party system has an independent strategy, resulting in a large development workload and operation and maintenance difficulty, which is not conducive to the later iteration and expansion of the software operation system. The method is applied to the software operation system and the third-party system. The software operation system can be a SaaS system. Figure 1 FIG. 1 is a first schematic diagram of a multi-source heterogeneous data processing method provided by an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0065] S100. Define the transmission mode and data format supported by the data access of the software operation system, and obtain multi-source heterogeneous data from a third-party system based on the transmission mode and data format.

[0066] In the embodiment of the present invention, the supported transmission modes and data formats are defined through the plug-in mode, which supports unlimited expansion while supporting common protocols and general data transmission modes of the transport layer and application layer. The transmission modes supported by data access include but are not limited to one or more of the http protocol, transport layer protocol, intermediate library, message queue and text data.

[0067] Optionally, the transmission mode supporting the http protocol realizes the interaction between systems by providing an application program interface restful api interface based on the state transmission, and realizes the external provision of interface services through the combined annotation @RestController annotation provided by the open source framework Spring framework, and provides corresponding interface documentation based on the interface platform YAPI. For the application programming interface provided by the third-party system, the client is defined by the client object httpclient in the embodiment of the present invention, and the request is dynamically assembled through the uniform resource locator URL, port and authentication information configured in the interface to obtain data.

[0068] Optionally, the transmission method of the transport layer protocol is supported to communicate in the local area network through the transmission control protocol TCP and the user datagram protocol UDP, so as to realize the transmission of instructions and the exchange of data, and then by creating a network server framework instance, define the IP address and port number for providing services to the outside world, and configure it in the front-end interface for use by third-party systems. At the same time, it can also support the definition of a socket-based client tool Socket Client instance as a client to obtain data from the server.

[0069] Optionally, the transmission method based on the intermediate library is a common way to define data interaction between systems. The writing and reading of data are completely separated and do not affect each other. First, the configuration information of the intermediate library is defined on the front end through the JavaScript framework vue3. The configuration information includes but is not limited to the type, driver, address, account and password of the database. Click Test Connection and Save Connection to verify and save the accuracy of the data configuration respectively. Then, through the obtained database account, read the metadata information, which includes but is not limited to table information, field information and table data, and the metadata information can be used for data conversion.

[0070] Optionally, the message queue transmission method is supported by adding the conventional open source message broker software RabbitMQ dependency in the project object model file pom.xml in the project management tool software Maven, and dynamically defining the listener interface MQMessageListener for processing messages for monitoring objects according to the message middleware type, message propagation method, consumer group and other information provided by the third-party system, and using the message listener onMessage method for data processing.

[0071] Optionally, support the transmission of text data. Historical stock data is generally archived and stored in local disks and the cloud. The embodiment of the present invention supports access to data stored in text types, including but not limited to csv, Excel, txt, and log types, mainly through the integrated object storage server MinIO. First, the configuration information of the object storage server MinIO is defined on the front end through the JavaScript framework vue3. The configuration information includes but is not limited to the address and port of the proxy server, the uniform resource locator URL, the basic package of the access service, and the storage container. According to the configuration information, file information similar to the effect of a remote folder can be seen on the interface, and the text data is read through the class StreamReader that reads text files for subsequent data processing.

[0072] S200: Process multi-source heterogeneous data.

[0073] A large amount of multi-source heterogeneous data is accessed through the data access transmission method, but these multi-source heterogeneous data cannot be used directly, so data processing is required. Figure 2 is a second schematic diagram of the multi-source heterogeneous data processing method provided by an embodiment of the present invention. Figure 2 As shown, data processing includes the following steps:

[0074] S210: Parse the data according to the data conversion protocol and encryption and decryption rules configured in the third-party system.

[0075] In the embodiment of the present invention, from the perspective of security, the data transmitted by the third-party system is not necessarily transmitted in plain text, so the data needs to be parsed according to the data conversion protocol and encryption and decryption rules configured in the third-party system. Figure 3 FIG. 1 is a first schematic diagram of a multi-source heterogeneous data processing method provided by an embodiment of the present invention. Figure 3 As shown, data parsing includes the following steps:

[0076] S211. Obtain configuration information of a third-party system according to each data source of multi-source heterogeneous data.

[0077] The configuration information of the third-party system is obtained according to various data sources. The configuration information includes but is not limited to encryption and decryption methods and key information. If the information cannot be obtained, it is assumed that the data transmitted by the third-party system is transmitted in plain text.

[0078] S212. Obtain an encryption algorithm instance object according to the configuration information.

[0079] S213. Decrypt the encrypted data in each data source according to the encryption algorithm instance object to obtain plaintext data.

[0080] The encryption algorithm instance object decrypts the encrypted data according to the public key (symmetric encryption), private key (asymmetric encryption) and summary information to obtain the plaintext data. If an exception occurs, the original data will be stored in the data cluster.

[0081] S220: Clean the data according to the data deduplication rules, necessary attributes and metadata definitions.

[0082] After data analysis, there are a lot of abnormal situations such as duplication, missing attributes and chaotic format in the data. The data is cleaned according to the deduplication rules, necessary attributes and metadata definitions of the data obtained from the data source. Figure 4 4 is a schematic diagram of a multi-source heterogeneous data processing method according to an embodiment of the present invention. Figure 4 As shown, data cleaning includes the following steps:

[0083] S221. De-duplicate the plaintext data according to de-duplication rules.

[0084] De-duplicate the plaintext data according to the defined de-duplication rules. For example, the de-duplication rule may be to take the mth piece of data as the representative data for the same data within n seconds.

[0085] S222. Filter the deduplicated plaintext data according to necessary attributes of the metadata.

[0086] According to metadata requirements, the plaintext data must have attributes. If these necessary attributes are not present in the plaintext data string, the plaintext data will be filtered out.

[0087] S223. Set format requirements for the filtered plain text data according to the regular expression, filter out difference data that does not meet the format requirements from the filtered plain text data, and obtain standard data.

[0088] The basic format requirements are set for the filtered plain text data according to the regular expression, and the difference data that does not meet the format requirements in the filtered plain text data is filtered out to obtain the standard data.

[0089] S230. Convert the cleaned data according to the third-party system and business type.

[0090] The data that has been cleaned basically meets the format requirements and basic attribute requirements. According to the third-party system and business type, all standard data obtained from the third-party system are uniformly converted into the data format specified by the software operation system. Figure 5 5 is a fifth schematic diagram of a multi-source heterogeneous data processing method provided by an embodiment of the present invention. Figure 5 As shown, data conversion includes the following steps:

[0091] S231. Obtain a metadata conversion configuration table according to a third-party system.

[0092] In an embodiment of the present invention, a metadata conversion configuration table is obtained according to a third-party system. Different configuration methods are provided according to different data formats of the third-party system, including but not limited to JSON format, special character segmentation format and value string format.

[0093] S232. Perform data conversion on the standard data according to the metadata conversion configuration table to obtain target data.

[0094] Optionally, if the configuration mode is in JSON format, the source property or target property is directly replaced, and the string is converted into a data object through JSON deserialization; if the configuration mode is in special character segmentation format, the values ​​are grouped through the string split method, the values ​​are obtained through the index index in the configuration, and the values ​​are set into the object properties through reflection; if the configuration mode is in value string format, the values ​​are obtained through string segmentation, and the values ​​are set into the object properties through reflection.

[0095] S240: Process the converted data.

[0096] For some data with higher requirements, data conversion is required for data processing, and the processing flow is mainly configured through the tool interface. Figure 6 6 is a schematic diagram of a multi-source heterogeneous data processing method according to an embodiment of the present invention. Figure 6 As shown, data processing includes at least one of the following steps:

[0097] S241, setting default values ​​for some attributes of the target data according to global default values;

[0098] S242. Supplement tenant information for target data based on tenants of third-party systems and software operating systems;

[0099] S243. Create data labels for target data based on data value.

[0100] Data is processed by one or more of the above-mentioned methods of setting default values, setting tenant information, and establishing data tags to obtain data that meets the requirements.

[0101] S300, hierarchically storing the processed multi-source heterogeneous data according to data attributes.

[0102] In an embodiment of the present invention, multi-source heterogeneous data are stored in layers according to data usage scenarios, read and write frequencies, and data importance. Figure 7 7 is a seventh schematic diagram of a multi-source heterogeneous data processing method provided by an embodiment of the present invention. Figure 7As shown, hierarchical storage of data includes the following steps:

[0103] S310: Store the configuration data and temporary data in the multi-source heterogeneous data after data processing in the cache middleware.

[0104] In the embodiment of the present invention, the data structure server Redis can be used as a cache middleware and deployed in sentinel mode. Optionally, the validity period of the configuration data is generally configured to be 7 days, and the synchronization can be modified according to the actual situation. The validity period of temporary data is generally configured to be 30 minutes, and it will automatically become invalid after expiration, thereby effectively saving memory.

[0105] S320: Establish a relational database, and store the basic data and business data in the multi-source heterogeneous data after data processing in the relational database.

[0106] The cache middleware can only temporarily store a small amount of configuration data and temporary data. By establishing a relational database, the data can be persisted to the disk. The relational database can not only store all basic data and configuration information, but also store the processed business data through the data persistence layer framework. In the embodiment of the present invention, the relational database includes but is not limited to Mysql, PostgreSQL, Oracle and other databases.

[0107] S330, building a data cluster, and storing the full life cycle data of the multi-source heterogeneous data after data processing in the data cluster.

[0108] In the embodiment of the present invention, the data cluster adopts the distributed system infrastructure Hadoop big data cluster. The Hadoop environment is implemented and deployed through the component CDH5.6, and then the data files of HDFS, the core component of the Hadoop ecosystem, are operated through the distributed column storage system HBase, and the big data retrieval is quickly performed based on the solr index library. Every time the multi-source heterogeneous data passes through a data processing link, a log data is inserted into HDFS, and a primary key is set for the business data. When querying, the primary key of the data is used for accurate retrieval, thereby realizing the full life cycle data management of multi-source heterogeneous data.

[0109] S400: Acquire post-monitoring configuration information of multi-source heterogeneous data after data processing, and trigger business logic operations according to the post-monitoring configuration information.

[0110] After data processing is completed, the post-monitoring configuration information is obtained according to the third-party system and business type, and the subsequent business logic operations are triggered by the post-monitoring configuration information. The business logic operations include but are not limited to application programming interface calls, scheduled task additions, and sending a message middleware message.

[0111] The multi-source heterogeneous data processing method provided by the embodiment of the present invention uses network protocols instead of third-party systems for data access, which greatly improves the code reuse rate, reduces the development workload, and realizes the basic functions while using redis, xml and message queues and other technical means to realize the processing of complex business, ensuring that the data is completely processed into data that meets business specifications. The attribute value replacement of the entity object is replaced by a configuration file, and the data conversion is realized in a higher abstract dimension. Data cleaning and data conversion can efficiently filter out valid data, improve the quality of data, and greatly improve the accuracy of conversion.

[0112] Example 2

[0113] Based on the same technical concept as the method embodiment of the above embodiment 1, the embodiment of the present invention provides a multi-source heterogeneous data processing device, which is applied to a software operation system and a third-party system. Figure 8 is a schematic diagram of the structure of a multi-source heterogeneous data processing device provided by an embodiment of the present invention, such as Figure 8 As shown, the multi-source heterogeneous data processing device 800 includes:

[0114] The data access module 810 is used to define the transmission mode and data format supported by the data access of the software operation system, and obtain multi-source heterogeneous data from a third-party system according to the transmission mode and data format.

[0115] The data processing module 820 is used to process multi-source heterogeneous data, including:

[0116] Parse data based on the data conversion protocol and encryption and decryption rules configured in the third-party system;

[0117] Clean the data according to the data deduplication rules, necessary attributes and metadata definitions;

[0118] Convert cleaned data according to third-party systems and business types;

[0119] Perform data processing on the data after data conversion.

[0120] The data storage module 830 is used to hierarchically store the processed multi-source heterogeneous data according to data attributes.

[0121] The post-trigger module 840 is used to obtain the post-monitoring configuration information of the multi-source heterogeneous data after data processing, and trigger the business logic operation according to the post-monitoring configuration information.

[0122] The source heterogeneous data processing device provided by the embodiment of the present invention can be infinitely expanded horizontally to achieve functions such as high performance, high availability and dynamic scaling. It has strong universality and configurability and can realize the visualization and configuration of data processing modules and the life cycle management of data.

[0123] It can be understood that the implementation method of the multi-source heterogeneous data processing method described in the above embodiment 1 is also applicable to this embodiment and can achieve the same technical effect, so it will not be repeated here.

[0124] Example 3

[0125] Based on the same concept, an embodiment of the present invention further provides an electronic device, Fig. 9 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig. 9 As shown, the electronic device 900 may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the steps of the multi-source heterogeneous data processing method described in the above embodiments. For example, it includes:

[0126] S100. Define the transmission mode and data format supported by the data access of the software operation system, and obtain multi-source heterogeneous data from the third-party system according to the transmission mode and data format;

[0127] S200, processing multi-source heterogeneous data;

[0128] S300, hierarchically storing the processed multi-source heterogeneous data according to data attributes;

[0129] S400: Acquire post-monitoring configuration information of multi-source heterogeneous data after data processing, and trigger business logic operations according to the post-monitoring configuration information.

[0130] The processor 910 may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0131] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0132] The memory 930 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] Example 4

[0134] Based on the same concept, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program, and the computer program includes at least one code segment, which can be executed by a main control device to control the main control device to implement the steps of the multi-source heterogeneous data processing method described in the above embodiments. For example, it includes:

[0135] S100. Define the transmission mode and data format supported by the data access of the software operation system, and obtain multi-source heterogeneous data from the third-party system according to the transmission mode and data format;

[0136] S200, processing multi-source heterogeneous data;

[0137] S300, hierarchically storing the processed multi-source heterogeneous data according to data attributes;

[0138] S400: Acquire post-monitoring configuration information of multi-source heterogeneous data after data processing, and trigger business logic operations according to the post-monitoring configuration information.

[0139] Based on the same technical concept, an embodiment of the present invention further provides a computer program, which is used to implement the above method embodiment when executed by a main control device.

[0140] The computer program may be stored in whole or in part on a computer-readable storage medium packaged with the processor, or may be stored in whole or in part on a memory not packaged with the processor.

[0141] Based on the same technical concept, an embodiment of the present invention further provides a processor, which is used to implement the above method embodiment. The above processor can be a chip.

[0142] In summary, the multi-source heterogeneous data processing method, device, electronic device and storage medium provided by the present invention use network protocols instead of third-party systems for data access, which greatly improves the reuse rate of codes, reduces development workload, and realizes basic functions while using technical means such as redis, xml and message queues to realize the processing of complex businesses, ensuring that data is completely processed into data that meets business specifications. The attribute value replacement of entity objects is replaced by configuration files, and data conversion is realized with a higher abstract dimension. Data cleaning and data conversion can efficiently filter out effective data, improve data quality, and greatly improve the accuracy of conversion. Relying on the statelessness of the present invention, it can be infinitely horizontally expanded to achieve functions such as high performance, high availability and dynamic scaling. It has strong universality and configurability, and can realize the visualization, configuration and life cycle management of data processing modules.

[0143] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0144] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source heterogeneous data processing method, the method is applied to software operation systems and third-party systems, characterized in that: The method comprises: Define the transmission mode and data format supported by the data access of the software operation system, and obtain multi-source heterogeneous data from the third-party system according to the transmission mode and the data format; The multi-source heterogeneous data is processed, including: Perform data analysis according to the data conversion protocol and encryption and decryption rules configured in the third-party system; Clean the data according to the data deduplication rules, necessary attributes and metadata definitions; Convert the cleaned data according to the third-party system and business type; Perform data processing on the data after data conversion; The processed multi-source heterogeneous data is stored in layers according to data attributes; Acquire post-monitoring configuration information of the multi-source heterogeneous data after the data processing, and trigger a business logic operation according to the post-monitoring configuration information.

2. The multi-source heterogeneous data processing method according to claim 1, characterized in that: The data parsing according to the data conversion protocol and encryption and decryption rules configured in the third-party system includes: Acquiring configuration information of the third-party system according to each data source of the multi-source heterogeneous data; Obtain an encryption algorithm instance object according to the configuration information; The encrypted data in each of the data sources is decrypted according to the encryption algorithm instance object to obtain plaintext data.

3. The multi-source heterogeneous data processing method according to claim 2, characterized in that: The data cleaning is performed according to the data deduplication rules, necessary attributes and metadata definitions, including: Deduplication of the plaintext data is performed according to a deduplication rule; Filter the deduplicated plaintext data according to the necessary metadata attributes; Format requirements are set for the filtered plaintext data according to the regular expression, and difference data that does not meet the format requirements are filtered out from the filtered plaintext data to obtain standard data.

4. The multi-source heterogeneous data processing method according to claim 3, characterized in that: The data conversion after data cleaning according to the third-party system and business type includes: Acquire a metadata conversion configuration table according to the third-party system; The standard data is converted according to the metadata conversion configuration table to obtain target data.

5. The multi-source heterogeneous data processing method according to claim 4, characterized in that: The data processing of the converted data includes at least one of the following steps: Setting default values ​​for some attributes of the target data according to global default values; Supplementing tenant information for the target data according to tenants of the third-party system and the software operation system; A data label is established for the target data according to the data value.

6. The multi-source heterogeneous data processing method according to claim 1, characterized in that: The step of storing the processed multi-source heterogeneous data in layers according to data attributes includes: storing the configuration data and temporary data in the multi-source heterogeneous data after the data processing in the cache middleware; Establishing a relational database, storing basic data and business data in the multi-source heterogeneous data after the data processing in the relational database; A data cluster is constructed, and the full life cycle data of the multi-source heterogeneous data after the data processing is stored in the data cluster.

7. The multi-source heterogeneous data processing method according to claim 1, characterized in that: The transmission modes supported by the data access include at least one or more of the following modes: http protocol, transport layer protocol, intermediate library, message queue and text data.

8. A multi-source heterogeneous data processing device, the device is applied to a software operation system and a third-party system, characterized in that: The device comprises: A data access module, used to define the transmission mode and data format supported by the data access of the software operation system, and to obtain multi-source heterogeneous data from the third-party system according to the transmission mode and the data format; A data processing module is used to process the multi-source heterogeneous data, including: Perform data analysis according to the data conversion protocol and encryption and decryption rules configured in the third-party system; Clean the data according to the data deduplication rules, necessary attributes and metadata definitions; Convert the cleaned data according to the third-party system and business type; Perform data processing on the data after data conversion; A data storage module, used for storing multi-source heterogeneous data after data processing in layers according to data attributes; The post-trigger module is used to obtain the post-monitoring configuration information of the multi-source heterogeneous data after the data processing, and trigger the business logic operation according to the post-monitoring configuration information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor executes the computer program to implement the multi-source heterogeneous data processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the multi-source heterogeneous data processing method according to any one of claims 1 to 7 is implemented.

Images