Multi-source data analysis processing method and system, electronic equipment and storage medium

Abstract

The invention discloses a multi-source data analysis processing method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining first data source information of a preset data source, so as to establish data connection; performing update monitoring on each preset data source, and judging whether the preset data source is updated or not; when it is determined that the preset data source is updated, calculating a performance index according to the preset resource information to generate a data acquisition script; performing data acquisition on a preset data source through the data acquisition script to obtain to-be-updated data; packaging the to-be-updated data in a preset format to obtain target packaged data including primary key information and content information; and according to the primary key information, updating the content information to a preset data set through the primary key mapping relationship to obtain a target data set, and responding to the query analysis request through the target data set. According to the embodiment of the invention, multi-dimensional data analysis across multiple types of databases can be realized, and the performance of data retrieval and query is improved. The method can be widely applied to the technical field of big data.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a multi-source data analysis and processing method, system, electronic device and storage medium. Background Art

[0002] With the rapid development of cloud computing and big data, more and more databases have emerged. In related technologies, when performing multidimensional data analysis, the data source needs to come from the same type of database, which makes it difficult to perform multidimensional analysis across multiple types of databases, and when selecting different dimensions, the query performance drops sharply.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a multi-source data analysis and processing method, system, electronic device and storage medium, which can realize multidimensional data analysis across multiple types of databases and effectively improve the performance of data retrieval and query.

[0005] To achieve the above purpose, an embodiment of the present application provides a multi-source data analysis and processing method, the method comprising the following steps:

[0006] Acquire first data source information of each preset data source, so as to establish a data connection with each preset data source through the first data source information;

[0007] Performing update monitoring on each of the preset data sources through the data connection to determine whether the preset data source has been updated;

[0008] When it is determined that the preset data source is updated, a corresponding performance indicator is calculated according to preset resource information of the preset data source, so as to generate a data collection script according to the performance indicator;

[0009] Collect data from the preset data source through the data collection script to obtain data to be updated;

[0010] Encapsulating the data to be updated in a preset format to obtain target encapsulated data; wherein the target encapsulated data includes primary key information and content information;

[0011] The content information is synchronously updated to a preset data set according to the primary key information through a primary key mapping relationship, and a target data set is obtained by integration, so as to respond to a query analysis request through the target data set.

[0012] In some embodiments, the calculating corresponding performance indicators according to the preset resource information of the preset data source to generate a data collection script through the performance indicators includes:

[0013] Dynamically calculating the data processing capability of the preset data source according to the preset resource information to obtain the performance indicator; wherein the preset resource information includes cluster size information and hardware resource information of the preset data source;

[0014] A preset data processing strategy is determined according to the performance indicator, so as to generate the data collection script through the preset data processing strategy; wherein the preset data processing strategy includes a computing resource pre-position strategy and a computing resource post-position strategy.

[0015] In some embodiments, determining a preset data processing strategy according to the performance indicator to generate the data collection script through the preset data processing strategy includes:

[0016] When it is determined that the performance indicator is greater than a preset indicator threshold, a first acquisition script is generated according to the computing resource pre-positioning strategy; wherein the first acquisition script is used to perform data extraction processing and pre-filtering combination processing through the preset data source;

[0017] Alternatively, when it is determined that the performance indicator is less than the preset indicator threshold, a second collection script is generated according to the computing resource post-positioning strategy; wherein the second collection script is used to perform data extraction processing.

[0018] In some embodiments, encapsulating the data to be updated in a preset format to obtain target encapsulated data includes:

[0019] Performing field attribution analysis on the data to be updated to obtain preset association information;

[0020] Determine corresponding primary key field data and non-primary key field data from the data to be updated according to the preset association information; wherein the primary key field data corresponds to the primary key information;

[0021] Converting the non-primary key field into a format using a preset data exchange format to obtain the content information; wherein the content information includes a preset format string;

[0022] The target encapsulated data is constructed according to the primary key field data and the preset format character string.

[0023] In some embodiments, the step of synchronously updating the content information into a preset data set through a primary key mapping relationship according to the primary key information to obtain a target data set includes:

[0024] Querying the association relationship between the target encapsulated data and the preset data set through a preset sorting algorithm according to the primary key field data;

[0025] Determine a formal data area according to the association relationship; wherein the formal data area includes an original data storage area corresponding to the primary key field data in the preset data set;

[0026] Integrate the preset format character strings to obtain a first data set, and store the first data set in a temporary data area;

[0027] When it is determined that the preset data set is not referenced, the linked list addresses of the formal data area and the temporary data area are exchanged to construct the target data set.

[0028] In some embodiments, responding to the query analysis request through the target data set includes:

[0029] When it is determined that the query analysis request is received, query and retrieve the target data list by using the primary key information according to the query analysis request;

[0030] Integrate the corresponding preset format character strings according to the target data list to obtain a target character string;

[0031] The target character string is encapsulated in a preset format to obtain a feedback data packet, and the feedback data packet is fed back to the corresponding query interface.

[0032] In some embodiments, the updating monitoring of each of the preset data sources through the data connection to determine whether the preset data source has been updated includes:

[0033] Dynamically monitor the data update status of the preset data source through a preset status indicator; wherein the preset status indicator includes source table update time or breakpoint information;

[0034] When it is determined that there is data update in the preset data source, the second data source information corresponding to the preset data source is written into the preset message queue, so as to perform incremental data collection on the preset data source according to the preset message queue in a preset sorting order; wherein the preset sorting order includes priority order and time order.

[0035] To achieve the above purpose, another aspect of the embodiment of the present application provides a multi-source data analysis and processing system, the system comprising:

[0036] A first module is used to obtain first data source information of each preset data source, so as to establish a data connection with each preset data source through the first data source information;

[0037] The second module is used to monitor the update of each of the preset data sources through the data connection to determine whether the preset data source has been updated;

[0038] The third module is used to calculate the corresponding performance index according to the preset resource information of the preset data source when it is determined that the preset data source is updated, so as to generate a data collection script according to the performance index;

[0039] The fourth module is used to collect data from the preset data source through the data collection script to obtain data to be updated;

[0040] The fifth module is used to encapsulate the data to be updated in a preset format to obtain target encapsulated data; wherein the target encapsulated data includes primary key information and content information;

[0041] The sixth module is used to synchronously update the content information into a preset data set according to the primary key information through a primary key mapping relationship, and integrate to obtain a target data set, so as to respond to the query analysis request through the target data set.

[0042] To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising:

[0043] at least one processor;

[0044] at least one memory for storing at least one program;

[0045] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0046] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0047] The embodiments of the present application include at least the following beneficial effects: The present application provides a multi-source data analysis and processing method, system, electronic device and storage medium. The scheme acquires the first data source information of each preset data source to establish a data connection with each preset data source through the first data edge line, thereby monitoring the update of each preset data source and determining whether the preset data source has been updated. Accordingly, when it is determined that the preset data source has been updated, the embodiment of the present invention calculates the corresponding performance index according to the preset resource information of the preset data source, generates a data collection script through the performance index, and then collects data from the preset data source through the data collection script to obtain the data to be updated. Furthermore, the embodiment of the present invention encapsulates the data to be updated in a preset format to obtain target encapsulated data including primary key information and content information, and then synchronizes the content information to the preset data set through the primary key mapping relationship according to the primary key information, and integrates to obtain the target data set, so as to respond to the query analysis request through the target data set, and realize multidimensional data analysis across multiple types of databases. It is easy to understand that the embodiments of the present invention can determine the data update status of each preset data source in real time by monitoring the update of the preset data source, and determine the corresponding data collection script according to the performance indicators of the preset data source, which can effectively improve the efficiency of data collection. At the same time, by packaging the collected update data in a preset format and integrating it into the target data set through the primary key mapping relationship, it is possible to provide data query services through the target data set, thereby effectively improving the efficiency of data retrieval queries and alleviating the problem of insufficient data timeliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a multi-source data analysis and processing method provided by an embodiment of the present invention;

[0049] Figure 2 It is a schematic flow chart of steps for calculating corresponding performance indicators according to preset resource information of a preset data source, so as to generate a data collection script through the performance indicators, provided by an embodiment of the present invention;

[0050] Figure 3 It is a schematic flow chart of steps for determining a preset data processing strategy according to performance indicators and generating a data collection script through the preset data processing strategy provided by an embodiment of the present invention;

[0051] Figure 4 It is a schematic flow chart of the steps of encapsulating the data to be updated in a preset format to obtain target encapsulated data provided by an embodiment of the present invention;

[0052] Figure 5 It is a flowchart of steps for synchronously updating content information to a preset data set through a primary key mapping relationship according to primary key information, and integrating to obtain a target data set, provided by an embodiment of the present invention;

[0053] Figure 6 is a schematic diagram of a step flow of responding to a query analysis request through a target data set provided by an embodiment of the present invention;

[0054] Figure 7 It is a schematic flow chart of the steps of monitoring updates of various preset data sources through data connections to determine whether the preset data sources have been updated, provided by an embodiment of the present invention;

[0055] Figure 8 is a schematic diagram of a multi-source data analysis and processing architecture provided by an embodiment of the present invention;

[0056] Fig. 9 is a data processing flow chart of multi-source data analysis and processing provided by an embodiment of the present invention;

[0057] Fig.10 It is a structural diagram of a multi-source data analysis and processing system provided in an embodiment of the present application;

[0058] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0060] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0061] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following explanations.

[0064] Heterogeneous data sources: refers to data sources of different types, structures and formats, which may include relational databases, non-relational databases, text files, CSV files, Excel tables, API interfaces, sensor data, log files, pictures, audio and video, etc. Accordingly, due to the diversity of data sources, these data will face more difficulties in storage, processing and analysis, such as data integration, format conversion and data cleaning.

[0065] Database: is an organized collection of data that allows users to store, manage and retrieve data in an efficient and convenient way. Accordingly, a database is a collection of data that is obtained by organizing data according to a certain structure to facilitate subsequent queries and operations.

[0066] With the rapid development of cloud computing and big data, more and more databases have emerged. In related technologies, when performing multidimensional data analysis, the data source needs to come from the same type of database, which makes it difficult to perform multidimensional analysis across multiple types of databases, and when selecting different dimensions, the query performance drops sharply. In addition, traditional multidimensional data analysis is inflexible, and there are many constraints when business personnel select data information. In addition, the maintenance cost of multidimensional models is high, and the workload of model changes and modifications is large.

[0067] In view of this, a multi-source data analysis and processing method, system, electronic device and storage medium are provided in the embodiments of the present application. The scheme obtains the first data source information of each preset data source to establish a data connection with each preset data source through the first data edge line, thereby monitoring the update of each preset data source and determining whether the preset data source has been updated. Accordingly, when it is determined that the preset data source has been updated, the embodiment of the present invention calculates the corresponding performance index according to the preset resource information of the preset data source to generate a data collection script through the performance index, and then collects data from the preset data source through the data collection script to obtain the data to be updated. Furthermore, the embodiment of the present invention encapsulates the data to be updated in a preset format to obtain target encapsulated data including primary key information and content information, and then synchronously updates the content information to the preset data set through the primary key mapping relationship according to the primary key information, and integrates to obtain the target data set, so as to respond to the query analysis request through the target data set, and realize multidimensional data analysis across multiple types of databases, effectively improving the performance of data retrieval and query.

[0068] The multi-source data analysis and processing method provided in the embodiment of the present application relates to the field of big data technology. The multi-source data analysis and processing method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in the blockchain network; the software can be an application that implements the multi-source data analysis and processing method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] Figure 1 is an optional flowchart of the multi-source data analysis and processing method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S160.

[0071] Step S110: Acquire first data source information of each preset data source to establish a data connection with each preset data source through the first data source information.

[0072] Step S120: monitoring each preset data source for updates through data connection to determine whether the preset data source has been updated.

[0073] Step S130: When it is determined that the preset data source is updated, a corresponding performance indicator is calculated according to preset resource information of the preset data source, so as to generate a data collection script through the performance indicator.

[0074] Step S140: collect data from a preset data source through a data collection script to obtain data to be updated.

[0075] Step S150: The data to be updated is packaged in a preset format to obtain target packaged data, wherein the target packaged data includes primary key information and content information.

[0076] Step S160: Synchronously update the content information to the preset data set through the primary key mapping relationship according to the primary key information, integrate to obtain the target data set, and respond to the query analysis request through the target data set.

[0077] During the working process of this specific embodiment, the embodiment of the present invention first obtains the first data source information of each preset data source to establish a data connection with each preset data source through the first data source information. Specifically, each preset data source in the embodiment of the present invention is a heterogeneous data source. For example, the preset data source in the embodiment of the present invention includes a database data source, a message or a file data source. Correspondingly, the first data source information in the embodiment of the present invention refers to the basic information of each heterogeneous data source. For example, the first data source information of the database data source obtained in the embodiment of the present invention includes a database IP, a database port, a database name, a data table name, etc., the first data source information of the file data source includes a collection protocol, an IP address, a port number, and a file name, etc., and the first data source information of the message data source includes an IP address, a port number, and a service name, etc. The embodiment of the present invention establishes a data connection with each heterogeneous data source by obtaining the corresponding basic information of each heterogeneous data source. Then, after establishing the data connection, the embodiment of the present invention monitors the update of each preset data source to determine whether the preset data source has been updated. Specifically, the embodiment of the present invention monitors the data changes of each heterogeneous data source in real time to determine whether each heterogeneous data source has data changes, that is, data updates. Accordingly, when there is a data change in a certain heterogeneous data source, that is, it is determined that there is an update in the preset data source, the corresponding updated data needs to be collected and synchronously updated. The embodiment of the present invention calculates the corresponding performance index based on the preset resource information of the preset data source to generate a data collection script through the performance index. Specifically, the preset resource information in the embodiment of the present invention refers to the relevant computing resource information of the preset data source where the data update occurs. Correspondingly, the performance index in the embodiment of the present invention refers to the data processing capability index of the preset data source where the data update occurs. The embodiment of the present invention analyzes and calculates the preset resource information of the preset data source where the data update occurs, thereby determining the current data processing capability of the preset data source, that is, the performance index, and then determining the corresponding data collection method and generating a data collection script. Among them, the data collection script in the embodiment of the present invention is dynamically determined according to the preset resource information of the corresponding preset data source, which can effectively improve the efficiency of data collection and processing.

[0078] Further, the embodiment of the present invention collects data from a preset data source through a data collection script to obtain data to be updated. Specifically, the data to be updated in the embodiment of the present invention refers to data that needs to be synchronized and updated to the target data set. Accordingly, the embodiment of the present invention collects data from a preset data source with data updates through a determined data collection method, i.e., a data collection script, and synchronizes the data to the cache through a corresponding collection engine, thereby obtaining the data to be updated. Next, the embodiment of the present invention encapsulates the data to be updated in a preset format to obtain target encapsulated data. Specifically, the target encapsulated data in the embodiment of the present invention includes primary key information and content information. Among them, after the embodiment of the present invention synchronizes the collected data to be updated to the cache, the data to be updated is encapsulated in the form of a key-value pair to obtain the target encapsulated data. Finally, the embodiment of the present invention synchronizes the content information to the preset data set through the primary key mapping relationship according to the primary key information to obtain the target data set through integration, thereby responding to the query analysis request through the target data set. Specifically, the embodiment of the present invention determines the association relationship between the target encapsulated data and the information in the existing memory, that is, the association relationship with the data in the preset data set, through the primary key information in the target encapsulated data and the corresponding primary key mapping relationship, so as to synchronously update the corresponding content information to the preset data set, and obtain the target data set by serially connecting and integrating the content information of multiple different dimensions and indicators. Among them, in the embodiment of the present invention, the target data set is stored in a distributed cache, and data support is provided to the upper part through the target data set. When a query analysis request is received, a data query is performed through the target data set to obtain a data query result, which effectively improves the performance of data retrieval and query. Accordingly, the embodiment of the present invention can effectively reduce the impact of data updates on queries by storing the data to be updated of each preset data source in columnar form through the primary key, and associating the data in the form of a linked list.

[0079] It should be noted that the implementation of the present invention supports cross-network multi-source heterogeneous data extraction and processing, extracts data through a variety of heterogeneous data source extraction components, obtains performance indicators of the front-end library based on the algorithm, implements conditional filtering, digital calculation and other operations in the database, and collects the result data through a unified collection engine. The location information corresponding to each data source is cached in the memory. Among them, the embodiment of the present invention supports but is not limited to the following database resources for data acquisition: Oracle, GP (Greenplum), TeraData, DB2, MySQL, PostgreSQL, Hive, Hbase, Hadoop, Xcloud, Dameng database and other data. At the same time, the embodiment of the present invention supports but is not limited to the following file service or message service resources for data acquisition: HDFS, FTP / SFTP, Kafka. Accordingly, the embodiment of the present invention supports but is not limited to the following methods for obtaining data: Socket, crawler, WebService, JDBC, front-end collection, etc.

[0080] Reference Figure 2 In some embodiments of the present invention, the corresponding performance indicators are calculated according to the preset resource information of the preset data source to generate a data collection script through the performance indicators, including but not limited to the following steps:

[0081] Step S210: dynamically calculating the data processing capability of the preset data source according to the preset resource information to obtain a performance index, wherein the preset resource information includes cluster size information and hardware resource information of the preset data source.

[0082] Step S220: Determine a preset data processing strategy according to the performance index to generate a data collection script through the preset data processing strategy, wherein the preset data processing strategy includes a computing resource pre-position strategy and a computing resource post-position strategy.

[0083] In this specific embodiment, the embodiment of the present invention first dynamically calculates the data processing capacity of the preset data source according to the preset resource information to obtain the corresponding performance index. Specifically, the preset resource information in the embodiment of the present invention includes the cluster scale information and hardware resource information of the preset data source. Among them, the cluster scale information in the embodiment of the present invention refers to the cluster scale of the corresponding preset data source, such as N servers. Correspondingly, the hardware resource information in the embodiment of the present invention refers to the hardware conditions of the preset data source, such as CPU, memory, and disk. The embodiment of the present invention dynamically calculates the data processing capacity of the preset data source through the cluster scale information and hardware resource information of the corresponding preset data source, thereby obtaining the current computing processing capacity of the preset data source, that is, the performance index. Exemplarily, when the cluster scale of the preset data source is N, the CPU core is K, the memory is M GB, and the disk is SAS (the disk is divided into three levels: flash, SAS, and SATA, corresponding to 1, 0.5, and 0.3 respectively), the processing capacity calculation formula is shown in the following formula (1):

[0084] C=N*(30%*K / 100+50%*M / 512+20%*0.5) (1)

[0085] Furthermore, the embodiment of the present invention determines the corresponding preset data processing strategy based on the calculated computing processing capacity, that is, the performance index, and generates a data collection script through the preset data processing strategy. Specifically, the preset data processing strategy in the embodiment of the present invention includes a computing resource pre-placement strategy and a computing resource post-placement strategy. Among them, the computing resource pre-placement strategy refers to pre-placing data computing processing to a preset data source for processing, such as database filtering. Correspondingly, the computing resource post-placement strategy refers to post-placing data computing processing, that is, filtering and calculating after the data is extracted. The embodiment of the present invention determines the corresponding preset data processing strategy through the performance index obtained by dynamic calculation, and then generates the corresponding data collection script, which can effectively improve the efficiency of data collection and extraction.

[0086] Combination Figure 2 , refer to Figure 3 In some embodiments of the present invention, a preset data processing strategy is determined according to the performance indicator to generate a data collection script through the preset data processing strategy, including but not limited to the following steps:

[0087] Step S310: When it is determined that the performance index is greater than the preset index threshold, a first acquisition script is generated according to the computing resource pre-positioning strategy, wherein the first acquisition script is used to perform data extraction processing and pre-filtering combination processing through a preset data source.

[0088] Step S320: When it is determined that the performance index is less than the preset index threshold, a second collection script is generated according to the computing resource post-positioning strategy, wherein the second collection script is used to perform data extraction processing.

[0089] In this specific embodiment, the embodiment of the present invention generates a corresponding data collection script by comparing the calculated performance index with the preset index threshold. Specifically, when it is determined that the performance index is greater than the preset index threshold, the embodiment of the present invention generates a first collection script according to the computing resource pre-position strategy. Alternatively, when it is determined that the performance index is less than the preset index threshold, the embodiment of the present invention generates a second collection script according to the computing resource post-position strategy. Among them, the first collection script in the embodiment of the present invention is used to perform data extraction processing and pre-filter combination processing through a preset data source, and the second collection script is used to perform data extraction processing. It is easy to understand that when the computing resource pre-position strategy is adopted, the embodiment of the present invention pre-places the data filtering method (database filtering), that is, the computing resources of the preset data source are used to perform data filtering combination processing after data extraction. For example, the embodiment of the present invention constructs a data filtering component and a combination component, pre-filters, and splices the filtering conditions and the combination conditions into corresponding SQL statements for filtering and combination. Correspondingly, when the computing resource post-position strategy is adopted, the collection engine in the embodiment of the present invention only extracts data, and then filters the data after extracting it. It should be noted that in some embodiments of the present invention, the interface layer and file data processing capabilities are 0, and a computing resource post-positioning strategy is required for processing.

[0090] It should be noted that the filtering conditions in the embodiment of the present invention support the following types: data integration is based on business needs, from multiple data sources, each source table extracts some fields, calculates through configuration and regularization, and combines into new data items. The data set is realized by associating N tables, and the fields are taken from some fields of each associated table; the multi-table association method supports internal association, external association, and full association; the association supports expression, filtering, aggregation, connection, sorting, and union. The associated table can restrict conditions; the field names of the table associations can be inconsistent, and multiple fields can be associated at the same time; the field name of the target field can be inconsistent with the source field name, and the source field can be renamed; the source field can be simply converted to generate the target field; a configurable data conversion method is provided, including: string conversion, field assignment, time type conversion, data comparison and translation, mathematical operation, ID card format conversion, etc.; a custom function package is provided, including: 50 commonly used functions such as time operation package and character operation package.

[0091] Reference Figure 4 In some embodiments of the present invention, the data to be updated is packaged in a preset format to obtain target packaged data, including but not limited to the following steps:

[0092] Step S410: Perform field attribution analysis on the data to be updated to obtain preset association information.

[0093] Step S420: Determine corresponding primary key field data and non-primary key field data from the data to be updated according to the preset association information, wherein the primary key field data corresponds to the primary key information.

[0094] Step S430: convert the non-primary key fields into a format using a preset data exchange format to obtain content information, wherein the content information includes a preset format string.

[0095] Step S440: constructing target encapsulated data according to the primary key field data and the preset format string.

[0096] In this specific embodiment, the embodiment of the present invention first performs field attribution analysis on the data to be updated to obtain preset association information, and then determines the corresponding primary key field data and non-primary key field data from the data to be updated according to the preset association information. Specifically, after synchronizing the data to the cache through the acquisition engine, the embodiment of the present invention obtains the associated primary key information of the data to be updated collected this time, that is, the preset association information, through field attribution analysis. The embodiment of the present invention parses and decomposes the fields in the data to be updated to determine the field with a unique identification function, that is, the primary key, so as to divide the data to be updated and determine the corresponding primary key field data and non-primary key field data. Among them, the primary key field data corresponds to the primary key information. In addition, the non-primary key field data corresponds to the content information, that is, the content data corresponding to the primary key. Further, the embodiment of the present invention converts the format of the non-primary key field through the preset data exchange format to obtain the content information. Specifically, the content information in the embodiment of the present invention includes a preset format string. For example, the preset data exchange format in the embodiment of the present invention can be Json format, and a Json string (preset format string) is generated separately for the non-primary key field of each table, and the data is associated in the form of a linked list, thereby reducing the impact of data update on the query. Finally, the embodiment of the present invention constructs the target encapsulated data according to the primary key field data and the preset format string. Specifically, the embodiment of the present invention encapsulates the target encapsulated data in the form of key-value pairs through the corresponding primary key field data and the preset format string, where key is the primary key information and value is the content information.

[0097] Reference Figure 5 In some embodiments of the present invention, content information is synchronously updated to a preset data set through a primary key mapping relationship according to primary key information, and a target data set is obtained by integration, including but not limited to the following steps:

[0098] Step S510: querying the association relationship between the target encapsulated data and the preset data set through a preset sorting algorithm according to the primary key field data.

[0099] Step S520: determining the formal data area according to the association relationship, wherein the formal data area includes the original data storage area corresponding to the primary key field data in the preset data set.

[0100] Step S530: Integrate the preset format character strings to obtain a first data set, and store the first data set in a temporary data area.

[0101] Step S540: When it is determined that the preset data set is not referenced, the linked list addresses of the formal data area and the temporary data area are exchanged to construct a target data set.

[0102] In this specific embodiment, the embodiment of the present invention first queries the association relationship between the target encapsulated data and the preset data set through a preset sorting algorithm according to the primary key field data. Specifically, the preset sorting algorithm in the embodiment of the present invention includes a quick sorting algorithm, a merge sorting algorithm, and a heap sorting algorithm. For example, the embodiment of the present invention constructs a quick sorting algorithm model to query the association relationship between each record information and the information in the existing memory, that is, the association relationship between the target encapsulated data and the preset data set. Among them, the embodiment of the present invention is connected in series in the form of a linked list, and at the same time, multiple Json strings of different dimensions and indicators are integrated into the target data set for query. Correspondingly, in the embodiment of the present invention, there are two copies of the data set, including temporary data (temporary data area) and formal data (formal data area), which are connected to the primary key in the form of a data linked list. The embodiment of the present invention determines the formal data area according to the association relationship. Specifically, the formal data area in the embodiment of the present invention includes the original data storage area corresponding to the primary key field data in the preset data set. Among them, the original data refers to the data before it is updated, that is, the historical data. The embodiment of the present invention obtains the storage position of the original data corresponding to the target encapsulated data in the preset data set through the association relationship query, thereby determining the formal data area. Next, the embodiment of the present invention integrates the preset format string to obtain the first data set, and stores the first data set in the temporary data area. Specifically, when there is data update, the temporary data area will integrate, convert, and produce a new data set. The embodiment of the present invention combines multiple preset format strings to form a new string, thereby constructing the first data set. For example, the embodiment of the present invention is stored in the form of Json strings, and multiple Json strings can be freely combined to produce new Json strings, thereby forming a flexible memory data set, which is provided to the upstream multidimensional analysis module for use. Accordingly, the embodiment of the present invention first stores the integrated first data set in the temporary data area. Finally, when it is determined that the preset data set is not referenced, the embodiment of the present invention exchanges the linked list addresses in the formal data area and the temporary data area, thereby constructing the target data set. Specifically, the embodiment of the present invention determines whether to perform the linked list address exchange by monitoring whether the preset data set is referenced. When it is detected that the preset data set is not referenced, the embodiment of the present invention exchanges the linked list addresses of the formal data area and the temporary data area, thereby realizing the data update of the preset data set and constructing a new data set, that is, the target data set. Among them, in the embodiment of the present invention, the time consumption of link list address exchange is controlled at millisecond level to ensure that the update process does not affect the query of the upstream system. In addition, in the embodiment of the present invention, the storage structure increases the memory address information, and the address information increases the IP address information and the memory address information. Multiple hosts can be formed into a federal system through the network.

[0103] Reference Figure 6 In some embodiments of the present invention, responding to a query analysis request through a target data set includes but is not limited to the following steps:

[0104] Step S610: When it is determined that a query analysis request has been received, query retrieval is performed based on the query analysis request using primary key information to obtain a target data list.

[0105] Step S620: integrating corresponding preset format strings according to the target data list to obtain a target string.

[0106] Step S630: encapsulate the target character string in a preset format to obtain a feedback data packet, and feed the feedback data packet back to the corresponding query interface.

[0107] In some embodiments of the present invention, when it is determined that a query analysis request has been received, the embodiment of the present invention performs query retrieval through the primary key information according to the query analysis request to obtain a target data list, and then integrates the corresponding preset format string according to the target data list to obtain the target string, thereby encapsulating the target string in a preset format to obtain a feedback data packet, and feeding it back to the query interface. Specifically, the embodiment of the present invention retrieves data access based on the primary key, performs data parsing and data calculation through a preset unpacking module, and feeds back the results to the upstream query interface. For example, when the query analysis request is received, the embodiment of the present invention performs a quick search based on the key value (primary key information) in the Json package through the retrieval module to obtain the relevant target data list. Then, the embodiment of the present invention parses and expands the key fields in Json through the calculation module, merges the data of multiple data sets based on the key, and parses, merges, calculates, and summarizes the field content (value) to obtain the target string. Finally, the embodiment of the present invention repackages the final result (target string) in Json format and feeds it back to the upstream query interface.

[0108] It should be noted that in the embodiment of the present invention, each data set is cached in the memory, stored in the Json format, sorted based on the unified user code, and other information that needs to be queried is sealed in the value in the Json format. When providing query services to the outside, the activity meets the conditions that need to be queried, first filters the target user list (target data list) in multiple data sets, and associates the data that need to be merged in the form of pointers to ensure that all attributes of the unified user can be quickly obtained. Based on the query conditions, the values ​​of each data set are unpacked, combined, calculated, and summarized, and finally the result information is repackaged into a Json string and provided to the data requester. After retrieval, the state can be cached within a certain time range to improve the efficiency of the query. At the same time, the embodiment of the present invention can be more than 5 times faster than direct query through Json format caching.

[0109] Reference Figure 7In some embodiments of the present invention, updating monitoring is performed on each preset data source through a data connection to determine whether the preset data source has been updated, including but not limited to the following steps:

[0110] Step S710: Dynamically monitor the data update status of the preset data source through a preset status indicator, wherein the preset status indicator includes source table update time or breakpoint information.

[0111] Step S720: When it is determined that the preset data source has data update, the second data source information corresponding to the preset data source is written into the preset message queue, so as to collect incremental data for the preset data source according to the preset message queue in a preset sorting order, wherein the preset sorting order includes priority order and time order.

[0112] In this specific embodiment, the embodiment of the present invention first dynamically monitors the data update status of the preset data source through a preset status indicator. Specifically, the preset status indicator in the embodiment of the present invention includes the source table update time or endpoint location information. Among them, the source table update time refers to the last update time of the source table recorded in the database system table, and the breakpoint information refers to the pause or failure point in the data synchronization process during the data processing or synchronization process. Correspondingly, the embodiment of the present invention specifies the incremental fields of each table in the process of setting the data source table through the incremental time standard and the breakpoint marking method, and provides a breakpoint information recording function for message data, and provides incremental real-time synchronization of data. Then, when it is determined that there is data update in the preset data source, the embodiment of the present invention writes the second data source information corresponding to the preset data source into the preset message queue, so as to perform incremental data collection for the preset data source according to the preset message queue in a preset sorting order. Specifically, the embodiment of the present invention reads the last update time of the source table recorded in the database system table through a polling thread, determines whether the data of the data table has changed, writes the changed data source information into the preset message queue, and the main thread reads the preset message queue information, and then re-collects the data. For example, the last time of the Oracle database record table is the dba_objects system table, and the query statement is as follows:

[0113] select t.last_ddl_time,t.*from dba_objects t

[0114] Where t.owner="AND t.object_name=";

[0115] The update time of the GP database library record table is the pg_stat_operations table. The query statement is as follows:

[0116] Select*frompg_stat_last_operation;

[0117] The update time of the MySQL database record table is `information_schema`.`TABLES`, and the query statement is as follows:

[0118] SELECT`TABLE_NAME`,`UPDATE_TIME`FROM

[0119] `information_schema`.`TABLES`WHERE

[0120] `information_schema`.`TABLES`.`TABLE_SCHEMA`="AND

[0121] `information_schema`.`TABLES`.`TABLE_NAME`=";

[0122] In addition, other incremental collection using the time field is performed when the system table cannot be obtained. Among them, the embodiment of the present invention obtains log information and incremental identification to obtain whether the current data collection requires increments and breakpoint positions. Conventionally, the time field is used for incremental collection, and the data obtained is the time span data from the end time of the last collection to the current time, and the time range is used as the condition for incremental collection. At the same time, the preset sorting order in the embodiment of the present invention includes priority order and time order. Among them, the preset message queue provides collection priority settings, which are divided into three types: high priority, medium priority, and low priority. The priorities of the same type are sorted in chronological order. The collection engine dispatches tasks based on the two dimensions of priority and time order. The total number of collection tasks is configurable, thereby ensuring that high priority is collected first when network resources are certain.

[0123] The following is a detailed description and explanation of the solution of the embodiment of the present invention in conjunction with a specific application example of multi-source data analysis and processing:

[0124] For example, Figure 8 As shown, Figure 8This is a schematic diagram of the multi-source data analysis and processing architecture of an embodiment of the present invention. Specifically, the data layer is carried by different databases, and the corresponding data is obtained through the data acquisition engine. The data integration engine will build a built-in data set in the memory, map it to different physical tables of different databases, and obtain the data changes in a regular manner in the form of tasks, synchronize it to the corresponding data set, and provide it to the upper-level data analysis. At the same time, the embodiment of the present invention provides specific support types for multiple sources, covering traditional relational databases, distributed databases, file type data sources, and interface-based message type data sources. In addition, the embodiment of the present invention combines technologies such as multi-source automatic adaptation components, built-in data sets, rapid data updates, and graphical analysis of charts, and combines visual matching pages and analysis pages to enable business personnel to quickly get started and meet the needs of self-service multi-dimensional analysis.

[0125] Accordingly, refer to Fig. 9 , Fig. 9 The data processing flow chart of the multi-source data analysis and processing of the embodiment of the present invention is shown. Specifically, the embodiment of the present invention first obtains the basic information of the data source of each heterogeneous data source to establish a data connection with each heterogeneous data source, so as to be able to access the data in each heterogeneous data source. Accordingly, the embodiment of the present invention determines whether the current data collection needs to be incrementally collected by obtaining the incremental identification or the address offset information of the last completed collection, and determines the location to start the collection. Further, the embodiment of the present invention dynamically calculates the computing power of each heterogeneous data source, such as calculating the processing power of the heterogeneous data source according to the cluster scale and hardware conditions (CPU, memory, disk) of the heterogeneous data source, and obtains the corresponding performance indicators, and then executes different collection strategies according to the performance indicators, such as computing resources pre-positioning or computing resources post-positioning, so as to dynamically generate the corresponding data collection script. Then, the embodiment of the present invention synchronizes the data to the cache through the collection engine according to the generated data collection script, obtains the primary key through homology analysis, and performs format conversion. Accordingly, the embodiment of the present invention integrates the format-converted data with the synchronized table through the associated field (primary key) to generate a memory cache data set to provide external query services. At the same time, the embodiment of the present invention dynamically monitors the change of data source through polling thread, determines whether there is data change in each heterogeneous data source, and re-triggers the collection process when it is determined that there is data change.

[0126] See also Fig.10 The embodiment of the present application also provides a multi-source data analysis and processing system, which can implement the above multi-source data analysis and processing method, and the system includes:

[0127] The first module 810 is used to obtain first data source information of each preset data source, so as to establish a data connection with each preset data source through the first data source information.

[0128] The second module 820 is used to monitor the update of each preset data source through the data connection to determine whether the preset data source is updated.

[0129] The third module 830 is used to calculate the corresponding performance index according to the preset resource information of the preset data source when it is determined that the preset data source is updated, so as to generate a data collection script through the performance index.

[0130] The fourth module 840 is used to collect data from a preset data source through a data collection script to obtain data to be updated.

[0131] The fifth module 850 is used to encapsulate the data to be updated in a preset format to obtain target encapsulated data, wherein the target encapsulated data includes primary key information and content information.

[0132] The sixth module 860 is used to synchronously update the content information into the preset data set according to the primary key information through the primary key mapping relationship, integrate to obtain the target data set, and respond to the query analysis request through the target data set.

[0133] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0134] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above multi-source data analysis and processing method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0135] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0136] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0137] The processor 910 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0138] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the multi-source data analysis and processing method of the embodiment of this application;

[0139] Input / output interface 930, used to implement information input and output;

[0140] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0141] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0142] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0143] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the multi-source data analysis and processing method described above is implemented.

[0144] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0145] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0146] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0147] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0148] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0149] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0150] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0151] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0153] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0156] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A multi-source data analysis and processing method, characterized in that: The method comprises the following steps: Acquire first data source information of each preset data source, so as to establish a data connection with each preset data source through the first data source information; Performing update monitoring on each of the preset data sources through the data connection to determine whether the preset data source has been updated; When it is determined that the preset data source is updated, a corresponding performance indicator is calculated according to preset resource information of the preset data source, so as to generate a data collection script according to the performance indicator; Collect data from the preset data source through the data collection script to obtain data to be updated; Encapsulating the data to be updated in a preset format to obtain target encapsulated data; wherein the target encapsulated data includes primary key information and content information; The content information is synchronously updated to a preset data set according to the primary key information through a primary key mapping relationship, and a target data set is obtained by integration, so as to respond to a query analysis request through the target data set.

2. The method according to claim 1, characterized in that The calculating corresponding performance indicators according to the preset resource information of the preset data source to generate a data collection script through the performance indicators includes: Dynamically calculating the data processing capability of the preset data source according to the preset resource information to obtain the performance indicator; wherein the preset resource information includes cluster size information and hardware resource information of the preset data source; A preset data processing strategy is determined according to the performance indicator, so as to generate the data collection script through the preset data processing strategy; wherein the preset data processing strategy includes a computing resource pre-position strategy and a computing resource post-position strategy.

3. The method according to claim 2, characterized in that The step of determining a preset data processing strategy according to the performance indicator to generate the data collection script through the preset data processing strategy includes: When it is determined that the performance indicator is greater than a preset indicator threshold, a first acquisition script is generated according to the computing resource pre-positioning strategy; wherein the first acquisition script is used to perform data extraction processing and pre-filtering combination processing through the preset data source; Alternatively, when it is determined that the performance indicator is less than the preset indicator threshold, a second collection script is generated according to the computing resource post-positioning strategy; wherein the second collection script is used to perform data extraction processing.

4. The method according to claim 1, characterized in that The step of encapsulating the data to be updated in a preset format to obtain target encapsulated data includes: Performing field attribution analysis on the data to be updated to obtain preset association information; Determine corresponding primary key field data and non-primary key field data from the data to be updated according to the preset association information; wherein the primary key field data corresponds to the primary key information; Converting the non-primary key field into a format using a preset data exchange format to obtain the content information; wherein the content information includes a preset format string; The target encapsulated data is constructed according to the primary key field data and the preset format character string.

5. The method according to claim 4, characterized in that The step of synchronously updating the content information into a preset data set through a primary key mapping relationship according to the primary key information and integrating to obtain a target data set includes: Querying the association relationship between the target encapsulated data and the preset data set through a preset sorting algorithm according to the primary key field data; Determine a formal data area according to the association relationship; wherein the formal data area includes an original data storage area corresponding to the primary key field data in the preset data set; Integrate the preset format character strings to obtain a first data set, and store the first data set in a temporary data area; When it is determined that the preset data set is not referenced, the linked list addresses of the formal data area and the temporary data area are exchanged to construct the target data set.

6. The method according to claim 4, characterized in that The responding to the query analysis request through the target data set includes: When it is determined that the query analysis request is received, query and retrieve the target data list by using the primary key information according to the query analysis request; Integrate the corresponding preset format character strings according to the target data list to obtain a target character string; The target character string is encapsulated in a preset format to obtain a feedback data packet, and the feedback data packet is fed back to the corresponding query interface.

7. The method according to claim 1, characterized in that The updating monitoring of each of the preset data sources through the data connection to determine whether the preset data source is updated includes: Dynamically monitor the data update status of the preset data source through a preset status indicator; wherein the preset status indicator includes source table update time or breakpoint information; When it is determined that there is data update in the preset data source, the second data source information corresponding to the preset data source is written into the preset message queue, so as to perform incremental data collection on the preset data source according to the preset message queue in a preset sorting order; wherein the preset sorting order includes priority order and time order.

8. A multi-source data analysis and processing system, characterized in that: The system comprises: A first module is used to obtain first data source information of each preset data source, so as to establish a data connection with each preset data source through the first data source information; The second module is used to monitor the update of each of the preset data sources through the data connection to determine whether the preset data source has been updated; The third module is used to calculate the corresponding performance index according to the preset resource information of the preset data source when it is determined that the preset data source is updated, so as to generate a data collection script according to the performance index; The fourth module is used to collect data from the preset data source through the data collection script to obtain data to be updated; The fifth module is used to encapsulate the data to be updated in a preset format to obtain target encapsulated data; wherein the target encapsulated data includes primary key information and content information; The sixth module is used to synchronously update the content information into a preset data set according to the primary key information through a primary key mapping relationship, and integrate to obtain a target data set, so as to respond to the query analysis request through the target data set.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images