Multi-data dynamic association merging method and device based on dynamic configuration, medium and equipment

Abstract

The invention discloses a multi-data dynamic association merging method and device based on dynamic configuration, a medium and equipment, and belongs to the technical field of computer data. The method comprises the following steps: inputting acquired data into a corresponding buffer area; constructing a hash table based on corresponding data; comparing the hash tables of the corresponding data, and outputting an intermediate result to an intermediate result output buffer area according to a comparison result; repeating the corresponding steps until the (n-1) th intermediate result data and the nth data are merged to obtain a final result, and outputting the final result to a final result output buffer area; and outputting the output data of the buffer area according to the final result, completing association and combination of multiple data, and obtaining a dynamic association and combination result, n being a positive integer greater than or equal to 2. The device, the medium and the equipment can be used for realizing the method. According to the method, real-time data can be rapidly collected, and the method has good fault-tolerant capability while supporting a large data volume.

Description

Technical Field

[0001] The present invention relates to the field of computer data technology, and in particular to a method, device, medium and equipment for dynamically associating and merging multiple data based on dynamic configuration. Background Art

[0002] In the prior art, the main disadvantages of the method for associating and merging multiple data for computer data include: lack of ability to authenticate secure access to services, and inability to provide security protection for access to services; failure to achieve rapid real-time data collection, and failure to support large amounts of data while having good fault tolerance; lack of fault recovery capability, that is, failure to rebuild the queue after the server crashes or restarts, and return reasonable status or abnormal information to the client based on the data loading status before the crash; failure to automatically perform quality analysis, and generate a data quality problem report for the processing process and unresolved problems; and failure to quickly analyze data to achieve a near-try-and-see effect. Summary of the invention

[0003] In view of this, the present invention provides a method, device, medium and equipment for dynamic association and merging of multiple data based on dynamic configuration, which can realize dynamic association and merging of multiple data based on dynamic configuration by constructing a data input buffer, an intermediate result output buffer, a final result output buffer, and hash table comparison, etc., and can realize rapid collection of real-time data. While supporting a large amount of data, it has good fault tolerance, and is therefore more suitable for practical use.

[0004] In order to achieve the first objective above, the technical solution of the method for dynamically associating and merging multiple data based on dynamic configuration provided by the present invention is as follows:

[0005] The present invention provides a method for dynamically associating and merging multiple data based on dynamic configuration, wherein the multiple data includes first data, second data, third data, and nth data, wherein n is a positive integer greater than or equal to 2, and the method for dynamically associating and merging multiple data includes the following steps:

[0006] Get the first data and the second data;

[0007] Inputting the first data into a first data input buffer, and inputting the second data into a second data input buffer;

[0008] Based on the first data input buffer, construct a hash table based on the first data; based on the second data input buffer, construct a hash table based on the second data;

[0009] Comparing the hash table of the first data and the hash table of the second data, and outputting a first intermediate result obtained by combining the first data and the second data to a first intermediate result output buffer according to the comparison result;

[0010] Based on the first intermediate result output buffer, construct a hash table based on the first intermediate result data of the first intermediate result buffer;

[0011] Get the third data;

[0012] Input the third data into a buffer;

[0013] Based on the third data input buffer, construct a hash table based on the third data;

[0014] Comparing the hash table of the first intermediate result data and the hash table of the third data, and outputting the second intermediate result obtained by combining the first intermediate result data and the third data to a second intermediate result output buffer according to the comparison result;

[0015] Repeat the above steps until the n-1th intermediate result data and the nth data are combined to obtain a final result and output it to the final result output buffer;

[0016] According to the output data of the final result output buffer, the association and merging of the first data, the second data, the third data, and the nth data are completed to obtain a dynamic association and merging result.

[0017] The method for dynamically associating and merging multiple data based on dynamic configuration provided by the present invention can also be further implemented by adopting the following technical measures.

[0018] Preferably, in the process of comparing the hash table of the first data and the hash table of the second data, comparing the hash table of the first intermediate result data and the hash table of the third data, and until comparing the hash table of the n-1th intermediate result data and the hash table of the nth data, the comparison is performed according to a set sliding window protocol.

[0019] Preferably, during the steps of obtaining the first data, the second data, the third data, and finally the nth data, a representational state transfer service is accessed, and a basic endpoint of the representational state transfer service is provided so that a service gateway can access the basic endpoint, wherein the basic endpoint is a unique feature, domain name, or path exposed to the outside via the service gateway.

[0020] Preferably, an identity authentication plug-in is configured in the service gateway, and the identity authentication plug-in is used to check the legitimacy of the token generated by the unified identity authentication service.

[0021] Preferably, before obtaining the first data, the second data, the third data, up to the nth data, the first data, the second data, the third data, up to the nth data are also preprocessed, which specifically includes the following steps in sequence: data extraction, data cleaning and conversion, data quality check, and data loading to obtain unified and complete subject data or data warehouse.

[0022] Preferably, the process of obtaining the first data, the second data, the third data, and finally the nth data is implemented by a distributed data flow engine, wherein the distributed data flow engine is implemented based on a distributed file system software, which is built on a distributed cluster and provides input, storage and output services for streaming data.

[0023] Preferably, the steps of obtaining the first data, the second data, the third data, and so on until the nth data also include the step of performing quality analysis on the data, and the quality analysis includes one or more of consistency check, missing value analysis, outlier analysis, effective value analysis, and validity analysis.

[0024] In order to achieve the above second purpose, the technical solution of the multi-data dynamic association and merging device based on dynamic configuration provided by the present invention is as follows:

[0025] The present invention provides a device for dynamically associating and merging multiple data based on dynamic configuration, wherein the multiple data includes first data, second data, third data, and nth data, wherein n is a positive integer greater than or equal to 2, and the device for dynamically associating and merging multiple data based on dynamic configuration includes:

[0026] A data acquisition unit, used for acquiring first data, second data, third data, and so on to nth data;

[0027] The data input buffer includes a first data input buffer, a second data input buffer, a third data input buffer, and an nth data input buffer, and is used to receive the first data, the second data, the third data, and the nth data respectively;

[0028] The intermediate result output buffer includes a first intermediate result output buffer, a second intermediate result output buffer, a third intermediate result output buffer, and an n-1th intermediate result output buffer, and is used to receive the first intermediate result, the second intermediate result, the third intermediate result, and the n-1th intermediate result, respectively;

[0029] A hash table construction unit, used to construct a hash table of corresponding data based on the data input buffer and the intermediate result output buffer;

[0030] A comparison unit, used to compare the hash tables of corresponding data;

[0031] The final result output buffer is used to receive the n-1th intermediate result data and the nth data and merge them to obtain the final result;

[0032] The dynamic association merging unit is used to complete the association merging of the first data, the second data, the third data, and up to the nth data according to the output data of the final result output buffer, so as to obtain a dynamic association merging result.

[0033] In order to achieve the third objective above, the technical solution of the computer-readable storage medium provided by the present invention is as follows:

[0034] The computer-readable storage medium provided by the present invention stores a multiple data dynamic association merging program based on dynamic configuration. When the multiple data dynamic association merging program based on dynamic configuration is executed by a processor, the steps of the multiple data dynamic association merging method based on dynamic configuration provided by the present invention are implemented.

[0035] In order to achieve the fourth objective, the technical solution of the electronic device provided by the present invention is as follows:

[0036] The electronic device provided by the present invention includes a memory and a processor, wherein a multiple data dynamic association merging program based on dynamic configuration is stored in the memory, and when the multiple data dynamic association merging program based on dynamic configuration is executed by the processor, the steps of the multiple data dynamic association merging method based on dynamic configuration provided by the present invention are implemented.

[0037] The method, device, medium and equipment for dynamically associating and merging multiple data based on dynamic configuration provided by the present invention can realize dynamic associating and merging multiple data based on dynamic configuration by constructing a data input buffer, an intermediate result output buffer, a final result output buffer, and hash table comparison, etc., and can realize rapid collection of real-time data, and have good fault tolerance while supporting a large amount of data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0039] Figure 1 A flowchart of the steps of a method for dynamically associating and merging multiple data based on dynamic configuration provided by an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the signal flow relationship between the functional modules in the multiple data dynamic association and merging device based on dynamic configuration provided by an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of a multi-data dynamic association and merging device structure based on dynamic configuration in a hardware operating environment provided by an embodiment of the present invention;

[0042] Figure 4 A flow chart of data collection for model units provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to solve the problems existing in the prior art, the present invention provides a method, device, medium and equipment for dynamically associating and merging multiple data based on dynamic configuration. By constructing a data input buffer, an intermediate result output buffer, a final result output buffer, and a hash table comparison, etc., the present invention can realize the dynamic associating and merging of multiple data based on dynamic configuration, can realize the rapid collection of real-time data, and has good fault tolerance while supporting a large amount of data, so that it is more suitable for practical use.

[0044] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the method, device, storage medium and electronic device for dynamically associating and merging multiple data based on dynamic configuration proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "embodiment" does not necessarily refer to the same embodiment. In addition, the features, structures, or characteristics in one or more embodiments may be combined in any suitable form.

[0045] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B. Specifically, it is understood that: A and B may be included at the same time, A may exist alone, or B may exist alone, and any of the above three situations may be met.

[0046] A method for dynamic association and merging of multiple data based on dynamic configuration

[0047] See attached Figure 1 The embodiment of the present invention provides a method for dynamically associating and merging multiple data based on dynamic configuration, wherein the multiple data includes first data, second data, third data, and nth data, wherein n is a positive integer greater than or equal to 2, and the method for dynamically associating and merging multiple data includes the following steps:

[0048] Step S1: Acquire the first data and the second data;

[0049] Step S2: inputting the first data into the first data input buffer, and inputting the second data into the second data input buffer;

[0050] Step S3: Based on the first data input buffer, construct a hash table based on the first data; based on the second data input buffer, construct a hash table based on the second data;

[0051] Step S4: comparing the hash table of the first data and the hash table of the second data, and outputting a first intermediate result obtained by combining the first data and the second data to a first intermediate result output buffer according to the comparison result;

[0052] Step S5: Based on the first intermediate result output buffer, construct a hash table of the first intermediate result data based on the first intermediate result buffer;

[0053] Step S6: Obtain the third data;

[0054] Step S7: input the third data into the buffer;

[0055] Step S8: constructing a hash table based on the third data based on the third data input buffer;

[0056] Step S9: comparing the hash table of the first intermediate result data and the hash table of the third data, and outputting the second intermediate result obtained by combining the first intermediate result data and the third data to the second intermediate result output buffer according to the comparison result;

[0057] Step S10: repeat the above steps until the n-1th intermediate result data and the nth data are combined to obtain the final result and output it to the final result output buffer;

[0058] Step S11: according to the output data of the final result output buffer, the association and merging of the first data, the second data, the third data, and up to the nth data are completed to obtain a dynamic association and merging result.

[0059] Among them, in the process of comparing the hash table of the first data and the hash table of the second data, comparing the hash table of the first intermediate result data and the hash table of the third data, until comparing the hash table of the n-1th intermediate result data and the hash table of the nth data, the comparison method is executed according to the set sliding window protocol.

[0060] Among them, in the steps of obtaining the first data, the second data, the third data, and finally the nth data, the representational state transfer service is accessed, and the basic endpoint of the representational state transfer service is provided, so that the service gateway can access the basic endpoint, wherein the basic endpoint is a unique feature, domain name or path exposed to the outside through the service gateway.

[0061] Among them, an identity authentication plug-in is configured in the service gateway, and the identity authentication plug-in is used to check the legitimacy of the token generated by the unified identity authentication service.

[0062] Among them, before obtaining the first data, the second data, the third data, to the nth data, it also includes preprocessing the first data, the second data, the third data, to the nth data, which specifically includes the following steps in sequence: data extraction, data cleaning and conversion, data quality check, and data loading to obtain unified and complete subject data or data warehouse.

[0063] The process of obtaining the first data, the second data, the third data, and finally the nth data is implemented through a distributed data flow engine, wherein the distributed data flow engine is implemented based on a distributed file system software, which is built on a distributed cluster and provides streaming data input, storage, and output services.

[0064] Among them, the steps of obtaining the first data, the second data, the third data, and finally the nth data also include the step of performing quality analysis on the data, and the quality analysis includes one or more of consistency test, missing value analysis, outlier analysis, effective value analysis, and effectiveness analysis.

[0065] Multiple data dynamic association and merging device based on dynamic configuration

[0066] See attached Figure 2 The embodiment of the present invention provides a device for dynamically associating and merging multiple data based on dynamic configuration, wherein the multiple data includes first data, second data, third data, and nth data, wherein n is a positive integer greater than or equal to 2, and the device for dynamically associating and merging multiple data based on dynamic configuration includes:

[0067] A data acquisition unit, used for acquiring first data, second data, third data, and so on to nth data;

[0068] The data input buffer includes a first data input buffer, a second data input buffer, a third data input buffer, and an nth data input buffer, and is used to receive the first data, the second data, the third data, and the nth data respectively;

[0069] The intermediate result output buffer includes a first intermediate result output buffer, a second intermediate result output buffer, a third intermediate result output buffer, and an n-1th intermediate result output buffer, and is used to receive the first intermediate result, the second intermediate result, the third intermediate result, and the n-1th intermediate result, respectively;

[0070] A hash table construction unit, used to construct a hash table of corresponding data based on a data input buffer and an intermediate result output buffer;

[0071] A comparison unit, used to compare the hash tables of corresponding data;

[0072] The final result output buffer is used to receive the n-1th intermediate result data and merge the nth data to obtain the final result;

[0073] The dynamic association merging unit is used to complete the association merging of the first data, the second data, the third data, and the nth data according to the output data of the final result output buffer, and obtain the dynamic association merging result.

[0074] Computer readable storage medium

[0075] The computer-readable storage medium provided by the present invention stores a multiple data dynamic association merging program based on dynamic configuration. When the multiple data dynamic association merging program based on dynamic configuration is executed by a processor, the steps of the multiple data dynamic association merging method based on dynamic configuration provided by the present invention are implemented.

[0076] Electronic devices

[0077] The electronic device provided by the present invention includes a memory and a processor. The memory stores a multiple data dynamic association merging program based on dynamic configuration. When the multiple data dynamic association merging program based on dynamic configuration is executed by the processor, the steps of the multiple data dynamic association merging method based on dynamic configuration provided by the present invention are implemented.

[0078] Reference Figure 3 , Figure 3 The present invention is a schematic diagram of a device structure for dynamically associating and merging multiple data based on dynamic configuration in a hardware operating environment according to an embodiment of the present invention.

[0079] like Figure 3 As shown, the multi-data dynamic association merging device based on dynamic configuration may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0080] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the device for dynamically associating and merging multiple data based on dynamic configuration, and may include more or less components than those shown in the figure, or combine certain components, or arrange the components differently.

[0081] like Figure 3 As shown, the memory 1005 as a storage medium may include an operating system, a data storage module, a network communication module, a user interface module, and a multi-data dynamic association and merging program based on dynamic configuration.

[0082] exist Figure 3 In the multi-data dynamic association and merging device based on dynamic configuration shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the multi-data dynamic association and merging device based on dynamic configuration of the present invention can be set in the multi-data dynamic association and merging device based on dynamic configuration, and the multi-data dynamic association and merging device based on dynamic configuration calls the multi-data dynamic association and merging program based on dynamic configuration stored in the memory 1005 through the processor 1001, and executes the multi-data dynamic association and merging method based on dynamic configuration provided by the embodiment of the present invention.

[0083] Example

[0084] Overall technical approach

[0085] Dynamic extraction of multiple data sources based on dynamic configuration is implemented based on service bus technology, which can support dynamic registration services of external business application systems and dynamically configure proxy rules. It has the ability to authenticate access to services, perform security protection on access to services, perform flow control on access to services, and perform operation and maintenance monitoring on access to services. The multi-source data automatic extraction component of the digital prototype model realizes the loading of structured data, semi-structured data, and unstructured data into the data management system, and realizes the pre-statistical analysis of the data. It is the entry point for digital prototype model data of aerospace models. Its main functions include data collection, streaming data processing, data quality analysis, real-time data analysis, data collection process control, etc.

[0086] Dynamic extraction of multiple data sources based on dynamic configuration is implemented based on service bus technology. Relying on service bus technology, it connects and connects external business application systems to achieve interoperability of service capabilities across technology platforms, application systems, and enterprise organizations. Through consistent organization and control of interoperable services, in the form of service APIs, the integration and interoperability of the internal and external business application systems of the digital prototype system are built. The service bus supports dynamic registration of services and can dynamically configure proxy rules. It consists of management APIs and reverse proxies.

[0087] By providing key extraction elements for various digital prototypes and specifying the protocols and data formats followed by service calls, the digital prototype system can provide consistent and configurable data extraction services for external heterogeneous business application systems. The system provides data extraction service registration and release, identity authentication, data collection, data analysis and process control technologies to achieve the extraction and aggregation of digital prototype models.

[0088] (1) Service registration and publishing: Service registration and publishing refers to registering service information on the service gateway so that the gateway can reverse proxy the service. For example, to access a RESTful service, you need to provide the base URL or endpoint of the service so that the service gateway can access it. You also need to provide the unique feature, domain name or path, that the service exposes to the outside through the gateway.

[0089] (2) Identity authentication: The service gateway of the digital prototype system is the only entrance for external system access. It cooperates with the unified authentication service to protect services that require authorized access and guide unauthorized access to unified authentication. At the same time, the service can support public services, that is, it allows the configuration of anonymous access rules. The service gateway has a built-in identity authentication plug-in. The service configured with the identity authentication plug-in will check the legitimacy of the token generated by the unified identity authentication service.

[0090] (2) Data extraction: Obtain the required multi-source heterogeneous data from a distributed environment. The data is resource data of different sources, formats, characteristics and properties, and files of different formats, including databases, XML documents, etc.

[0091] (3) Data cleaning and conversion: Data cleaning is to clean the extracted incomplete data, erroneous data, and duplicate data according to pre-designed rules, and to deal with some redundant and ambiguous data; data conversion mainly deals with null values, data splitting, data replacement, and data association to unify heterogeneous data formats. Before performing data ETL, it is necessary to formulate a complete data standard in advance to stipulate the unique and effective constraints for various types of data.

[0092] (4) Data quality check: Check the source data and cleaned data to ensure the consistency and correctness of the loaded data.

[0093] (5) Data loading: Data in the data buffer is directly loaded into the corresponding table of the database. Both full and incremental methods are supported. If it is full, the LOAD loading method is used. If it is incremental, the MERGE method is used to merge the data into the database according to business rules for index building processing.

[0094] Data Collection Technology

[0095] The business data of the model unit is the main source of platform data and is also one of the links in building a data warehouse. The data acquisition and scheduling subsystem collects the data from the distributed and heterogeneous data sources of the model unit, such as relational data and flat data files, and extracts them to the temporary intermediate layer for cleaning, conversion, integration, and loading into the data warehouse, and finally obtains a unified and complete subject database or data warehouse, which becomes the basis for online analytical processing and data mining. It includes support for multiple data format types such as database original backup collection, standard format file collection, file collection, and unstructured electronic documents, and supports automated and high-performance processing of collection.

[0096] After the model unit data is submitted to the collection and dispatch module, it will be processed by the dispatch center according to the type of data submitted and the attribute information of the data, such as unit information, data classification, collection time, etc. Figure 4 .

[0097] Model unit data has different processing procedures according to different classifications. It goes through processes such as data recovery, data migration, data processing, and data cleaning, and is stored in the original file storage, original library, isomorphic library, and standard library provided by the data storage management subsystem.

[0098] The data processing process is scheduled by the dispatch center. According to the preset configuration and information configured by the data administrator, different types of collected data are automatically processed according to the specified process, and high-performance support is provided for each process and each processing stage.

[0099] Streaming Data Processing Technology

[0100] The business data is extremely large. If it is simply migrated to the target database and then processed, the time and space costs are often extremely large. It is necessary to customize the development of stream processing functions according to the characteristics of the data. The streaming data processing function is similar to the publish / subscribe message system function. It provides distributed data streams, structured data streams, natural language processing and other processing technologies, which can realize real-time collection of feedback information, support large amounts of data and have good fault tolerance. It is mainly used to solve the problem of efficient processing and analysis of real-time data, and ensure that the streaming data that needs to be processed, cleaned, and converted is processed in time before persistence, thereby avoiding multiple scans of the database, reducing disk IO, and improving data processing efficiency.

[0101] The distributed data stream engine is implemented based on distributed file system software. It is built on a distributed cluster and provides high-throughput streaming data input, storage and output services. Distributed message queue technology is used to support the management of distributed data streams. Specific functions include: supporting data insertion, extraction, partitioning, replication, sorting, and deletion. In order to ensure data security, data is inserted into the queue and persistence operations are performed immediately. It supports the management of data production and consumption relationships, such as publishing, subscribing, and notifications; it supports queue management operations such as message queue creation, deletion, partition addition, and deletion; it has fault recovery capabilities, that is, the queue can be rebuilt after the server crashes or restarts, and it can return reasonable status or exception information to the client based on the data loading status before the crash.

[0102] When multiple data streams arrive at the same time, the data will be input into two different buffers first. Two buffers are selected to compare the merged KEY values. The comparison is performed according to the set sliding window protocol, and the intermediate results are compared with other buffers in the same way.

[0103] Data quality analysis technology

[0104] In the process of building a data warehouse for a business theme, data quality analysis is very important. It is necessary to discover and locate the types and problems of data that do not meet the requirements, and to be able to perform intelligent processing according to rules or statistical mining methods, and generate a data quality problem report for the processing process and unresolved problems. In general, it includes five parts: "consistency test", "missing value analysis", "outlier analysis", "validity analysis" and "data quality report".

[0105] Based on the data quality analysis module, data quality inspection rules are formulated for the process nodes that need to control the quality of the business data collection of the model unit, and data quality operators are formed. The process orchestration function is used to associate the data quality operators with the processing procedures that need to be inspected, and data quality exception handling is formulated. When the process passes through the quality inspection node, the inspection is performed. When the data quality does not meet the inspection rule requirements, the problem is recorded. Determine whether to perform exception handling based on the inspection rules. Generate data quality reports regularly based on the data quality inspection results.

[0106] Real-time data analysis technology

[0107] Real-time data analysis can quickly analyze model data to achieve near real-time results, which is of great significance for timely reflecting the value of model data. By establishing stream computing and real-time computing engines in the real-time computing layer through real-time data analysis, business needs such as data statistics, data fusion, and data labeling in a short time window can be met; by distributing the data received by the subscription distribution layer or the result data of real-time calculation to external systems, the pressure on the storage system caused by the external system obtaining massive data from the storage domain can be avoided.

[0108] The distributed real-time data analysis system provides the ability to process large-scale streaming data, a distributed computing framework, elastic distributed data sets, and a rich model that can quickly iterate data sets in memory multiple times to support complex data mining algorithms and graph computing algorithms. Through rich APIs and a high-speed memory-based computing engine, users can combine applications such as streaming processing, batch processing, and interactive queries.

[0109] Comparison table of technical effects between this embodiment and the prior art

[0110]

[0111] Possess the ability to authenticate secure access to services and provide security protection for access to services

[0112] The service gateway of the digital prototype system is the only entrance to the external system. It cooperates with the unified authentication service to protect the services that require authorized access and guide the unauthorized access to unified authentication. At the same time, the service can support public services, that is, it allows the configuration of anonymous access rules. The service gateway has a built-in authentication plug-in. The service configured with the authentication plug-in will check the legitimacy of the token generated by the unified authentication service.

[0113] Realize fast real-time data collection, support large data volumes and have good fault tolerance

[0114] The business data is extremely large. If it is simply migrated to the target database and then processed, the time and space costs are often extremely large. It is necessary to customize and develop stream processing functions according to the characteristics of the data. The streaming data processing function is similar to the publish / subscribe message system function. It provides distributed data streams, structured data streams, natural language processing and other processing technologies, which can realize real-time collection of feedback information, support large amounts of data and have good fault tolerance. It is mainly used to solve the problem of efficient processing and analysis of real-time data, and ensure that the streaming data that needs to be processed, cleaned, and converted is processed in time before persistence, thereby avoiding multiple scans of the database, reducing disk IO, and improving data processing efficiency.

[0115] The ability to quickly recover from device failures, that is, if the queue cannot be rebuilt after the server crashes or restarts, a reasonable status or abnormal information will be returned to the client based on the data loading status before the crash.

[0116] The distributed data stream engine is implemented based on distributed file system software. It is built on a distributed cluster and provides high-throughput streaming data input, storage and output services. Distributed message queue technology is used to support the management of distributed data streams. Specific functions include: supporting data insertion, extraction, partitioning, replication, sorting, and deletion. In order to ensure data security, when data is inserted into the queue, persistence operations will be performed immediately to support the management of data production and consumption relationships, such as publishing, subscribing, and notifications; support the creation and deletion of message queues, and queue management operations such as partition addition and deletion; have fault recovery capabilities, that is, the queue can be rebuilt after the server crashes or restarts, and can automatically perform quality analysis based on the data loading status before the crash, and generate data quality problem reports for the processing process and unresolved problems.

[0117] Based on the data quality analysis module, data quality inspection rules are formulated for the process nodes that need to control the quality of the business data collection of the model unit, and data quality operators are formed. The process orchestration function is used to associate the data quality operators with the processing procedures that need to be inspected, and data quality exception handling is formulated. When the process passes through the quality inspection node, the inspection is performed. When the data quality does not meet the inspection rule requirements, the problem is recorded. Determine whether to perform exception handling based on the inspection rules. Generate data quality reports regularly based on the data quality inspection results.

[0118] The model data can be quickly analyzed to achieve near real-time results, which is of great significance for timely reflecting the value of model data.

[0119] Real-time data analysis can quickly analyze model data to achieve near real-time results, which is of great significance for timely reflecting the value of model data. By establishing stream computing and real-time computing engines in the real-time computing layer through real-time data analysis, business needs such as data statistics, data fusion, and data labeling in a short time window can be met; by distributing the data received by the subscription distribution layer or the result data of real-time calculation to external systems, the pressure on the storage system caused by the external system obtaining massive data from the storage domain can be avoided.

[0120] The distributed real-time data analysis system provides the ability to process large-scale streaming data, a distributed computing framework, elastic distributed data sets, and a rich model that can quickly iterate data sets in memory multiple times to support complex data mining algorithms and graph computing algorithms. Through rich APIs and a high-speed memory-based computing engine, users can combine applications such as streaming processing, batch processing, and interactive queries.

[0121] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0122] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for dynamically associating and merging multiple data based on dynamic configuration, wherein the multiple data includes first data, second data, third data, and finally nth data, wherein: n is a positive integer greater than or equal to 2, characterized in that the multi-data association merging method comprises the following steps: Get the first data and the second data; Inputting the first data into a first data input buffer, and inputting the second data into a second data input buffer; Based on the first data input buffer, construct a hash table based on the first data; based on the second data input buffer, construct a hash table based on the second data; Comparing the hash table of the first data and the hash table of the second data, and outputting a first intermediate result obtained by combining the first data and the second data to a first intermediate result output buffer according to the comparison result; Based on the first intermediate result output buffer, construct a hash table based on the first intermediate result data of the first intermediate result buffer; Get the third data; Input the third data into a buffer; Based on the third data input buffer, construct a hash table based on the third data; Comparing the hash table of the first intermediate result data and the hash table of the third data, and outputting the second intermediate result obtained by combining the first intermediate result data and the third data to a second intermediate result output buffer according to the comparison result; Repeat the above steps until the n-1th intermediate result data and the nth data are combined to obtain a final result and output it to the final result output buffer; According to the output data of the final result output buffer, the association and merging of the first data, the second data, the third data, and the nth data are completed to obtain a dynamic association and merging result.

2. The method for dynamically associating and merging multiple data based on dynamic configuration according to claim 1, characterized in that: In the process of comparing the hash table of the first data and the hash table of the second data, comparing the hash table of the first intermediate result data and the hash table of the third data, until comparing the hash table of the n-1th intermediate result data and the hash table of the nth data, the comparison method is executed according to the set sliding window protocol.

3. The method for dynamically associating and merging multiple data based on dynamic configuration according to claim 1, characterized in that: In the process of obtaining the first data, the second data, the third data, and finally the nth data, a representational state transfer service is accessed, and a basic endpoint of the representational state transfer service is provided so that a service gateway can access the basic endpoint, wherein the basic endpoint is a unique feature, domain name, or path exposed to the outside through the service gateway.

4. The method for dynamically associating and merging multiple data based on dynamic configuration according to claim 3 is characterized in that: The service gateway is configured with an identity authentication plug-in, and the identity authentication plug-in is used to check the legitimacy of the token generated by the unified identity authentication service.

5. The method for dynamically associating and merging multiple data based on dynamic configuration according to claim 1, characterized in that: Before obtaining the first data, the second data, the third data, and the nth data, the first data, the second data, the third data, and the nth data are also preprocessed, which specifically includes the following steps in sequence: data extraction, data cleaning and conversion, data quality check, and data loading to obtain unified and complete subject data or data warehouse.

6. The method for dynamically associating and merging multiple data based on dynamic configuration according to claim 1, characterized in that: The process of obtaining the first data, the second data, the third data, and finally the nth data is implemented through a distributed data flow engine, wherein the distributed data flow engine is implemented based on a distributed file system software, which is built on a distributed cluster and provides streaming data input, storage and output services.

7. The method for dynamically associating and merging multiple data based on dynamic configuration according to claim 1, characterized in that: The steps of obtaining the first data, the second data, the third data, and finally the nth data also include the step of performing quality analysis on the data, and the quality analysis includes one or more of consistency check, missing value analysis, outlier analysis, effective value analysis, and effectiveness analysis.

8. A device for dynamically associating and merging multiple data based on dynamic configuration, wherein the multiple data includes first data, second data, third data, and so on to nth data, wherein: n is a positive integer greater than or equal to 2, characterized in that the multiple data dynamic association merging device based on dynamic configuration includes: A data acquisition unit, used for acquiring first data, second data, third data, and so on to nth data; The data input buffer includes a first data input buffer, a second data input buffer, a third data input buffer, and an nth data input buffer, and is used to receive the first data, the second data, the third data, and the nth data respectively; The intermediate result output buffer includes a first intermediate result output buffer, a second intermediate result output buffer, a third intermediate result output buffer, and an n-1th intermediate result output buffer, and is used to receive the first intermediate result, the second intermediate result, the third intermediate result, and the n-1th intermediate result, respectively; A hash table construction unit, used to construct a hash table of corresponding data based on the data input buffer and the intermediate result output buffer; A comparison unit, used to compare the hash tables of corresponding data; The final result output buffer is used to receive the n-1th intermediate result data and the nth data and merge them to obtain the final result; The dynamic association merging unit is used to complete the association merging of the first data, the second data, the third data, and up to the nth data according to the output data of the final result output buffer, so as to obtain a dynamic association merging result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a multiple data dynamic association merging program based on dynamic configuration. When the multiple data dynamic association merging program based on dynamic configuration is executed by the processor, the steps of the multiple data dynamic association merging method based on dynamic configuration described in any one of claims 1-7 are implemented.

10. An electronic device, characterized in that: It includes a memory and a processor, wherein a multiple data dynamic association merging program based on dynamic configuration is stored in the memory, and when the multiple data dynamic association merging program based on dynamic configuration is executed by the processor, the steps of the multiple data dynamic association merging method based on dynamic configuration described in any one of claims 1-7 are implemented.

Images