Data query method, device and equipment based on multi-source data and storage medium

Abstract

The embodiment of the invention provides a data query method and device based on multi-source data, equipment and a storage medium. According to the technical scheme provided by the embodiment of the invention, the data query request sent by the user side is acquired, the configuration file and the data query parameter are determined according to the data query request, and the data request parameter is determined based on the request construction rule and the data query parameter; and obtaining a data processing result from one or more preset data sources based on the data request parameter and the execution calling strategy, performing data analysis processing on the data processing result based on a response analysis rule to obtain a data query result, and sending the data query result to the user side, according to the method, the request construction rule, the execution calling strategy and the response analysis rule of the data source are defined through the configuration file, so that rapid calling and configuration of the multiple data sources can be realized, the tedious hard coding operation of docking the data sources can be effectively reduced, and the flexibility of multi-source data query is effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a data query method, apparatus, device and storage medium based on multi-source data. Background Art

[0002] With the development of computer technology and Internet technology, there are more and more forms of data storage, and it is often necessary to query, process and display information from multiple data sources. Different data sources can be various databases of the enterprise, application programming interfaces (API interfaces) provided by various services, background process services and file systems. And with the complexity and diversification of business, the number and types of data sources are also increasing. Traditional data integration and display methods are difficult to adapt to the rapidly changing needs of data sources.

[0003] At present, data query for multi-source data is generally based on hard-coded docking, that is, multiple data sources are connected through hard coding. However, every time a new data source is added, developers need to write new code to connect to the new data source. Every change in requirements requires manual modification of the data docking code and re-release of the online system. This type of data query cannot flexibly cope with the update of multi-source data, and the flexibility of multi-source data query is poor. Summary of the invention

[0004] The embodiments of the present application provide a data query method, apparatus, device and storage medium based on multi-source data to solve the technical problem of poor flexibility in data query of multi-source data in related technologies, and can effectively improve the flexibility of multi-source data query.

[0005] In a first aspect, an embodiment of the present application provides a data query method based on multi-source data, obtaining a data query request sent by a user terminal, and determining a configuration file and data query parameters according to the data query request, wherein the configuration file records a request construction rule, an execution call strategy, and a response parsing rule of one or more preset data sources, and the execution call strategy is used to record a call strategy for one or more preset data sources;

[0006] Determine data request parameters based on the request construction rule and the data query parameters;

[0007] Obtaining data processing results from one or more preset data sources based on the data request parameters and the execution call strategy;

[0008] The data processing result is subjected to data analysis processing based on the response analysis rule to obtain a data query result, and the data query result is sent to the user terminal.

[0009] In a second aspect, an embodiment of the present application provides a data query device based on multi-source data, including a request response module, a parameter determination module, a data acquisition module and a result determination module, wherein:

[0010] The request response module is configured to obtain a data query request sent by a user terminal, and determine a configuration file and data query parameters according to the data query request, wherein the configuration file records a request construction rule, an execution call strategy, and a response parsing rule of one or more preset data sources, and the execution call strategy is used to record a call strategy for one or more preset data sources;

[0011] The parameter determination module is configured to determine the data request parameters based on the request construction rule and the data query parameters;

[0012] The data acquisition module is configured to acquire data processing results from one or more preset data sources based on the data request parameters and the execution call strategy;

[0013] The result determination module is configured to perform data analysis processing on the data processing result based on the response analysis rule to obtain a data query result, and send the data query result to the user terminal.

[0014] In a third aspect, an embodiment of the present application provides a data query device based on multi-source data, including: a memory and one or more processors;

[0015] The memory is used to store one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the data query method based on multi-source data as described in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a non-volatile storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute the data query method based on multi-source data as described in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored in a computer-readable storage medium, and at least one processor of a device reads and executes the computer program from the computer-readable storage medium, so that the device performs the data query method based on multi-source data as described in the first aspect.

[0019] The embodiment of the present application obtains a data query request sent by a user terminal, determines a configuration file and data query parameters according to the data query request, determines data request parameters based on request construction rules and data query parameters, obtains data processing results from one or more preset data sources based on the data request parameters and execution call strategies, performs data parsing on the data processing results based on response parsing rules to obtain data query results, and sends the data query results to the user terminal. By defining the request construction rules, execution call strategies, and response parsing rules of the data source through the configuration file, multiple data sources can be quickly called and configured, which can effectively reduce cumbersome hard-coded operations for docking data sources and effectively improve the flexibility of multi-source data queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a data query method based on multi-source data provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of another data query method based on multi-source data provided by an embodiment of the present application;

[0022] Figure 3 This is a schematic diagram of a data processing result acquisition process provided by an embodiment of the present application;

[0023] Figure 4 It is a structural schematic diagram of a data query device based on multi-source data provided by an embodiment of the present application;

[0024] Figure 5 It is a structural diagram of a data query device based on multi-source data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The above process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The above process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0026] The data query method based on multi-source data provided in this application can be applied to data query scenarios of multiple data sources in systems such as big data analysis platforms and intelligent monitoring systems, such as various databases of the same docking enterprise, API interfaces provided by various services, background process services, file systems and other different data sources, thereby providing users with unified data access and processing capabilities. It aims to define the request construction rules, execution call strategies and response parsing rules of the data source through configuration files, realize the rapid calling and configuration of multiple data sources, and improve the flexibility of multi-source data query.

[0027] In the existing data query solutions for multi-source data, they are generally implemented based on hard-coded docking, middleware or data warehouse and BI (business intelligence) tools. In the data query solution based on hard-coded docking, each time a new data source is added, the developer needs to write new code to dock the new data source. Each change in requirements (such as field modification and addition) requires manual modification of the code and re-release of the online system. This method is not only inefficient, but also easy to introduce vulnerabilities and errors, and the system maintenance is difficult. For data query solutions based on middleware, middleware tools (such as ETL, Extract-Transform-Load) are generally used to extract, transform and load multiple data sources, and then uniformly display them in the display layer. Although middleware can realize the function of data integration, the data sources and data source request methods in actual applications are diverse, there are various uncertain and irregular data organization methods, and the configuration and use of middleware are relatively complex, requiring professionals to maintain the middleware, which increases the complexity of the system. The data query solution based on data warehouse and BI tools generally stores and manages data from multiple data sources in a unified manner by establishing a data warehouse, and combines BI tools to display and analyze data. However, the implementation cost of data warehouse and BI tools is high, the cycle is long, and the maintenance of data warehouse and BI tools is difficult. Based on this, a data query method based on multi-source data in an embodiment of the present application is provided to solve the technical problems of poor flexibility and difficult maintenance of existing multi-source data query.

[0028] Figure 1 A flow chart of a data query method based on multi-source data provided in an embodiment of the present application is given. The data query method based on multi-source data provided in an embodiment of the present application can be executed by a data query device based on multi-source data. The data query device based on multi-source data can be implemented by hardware and / or software and integrated in a data query device based on multi-source data.

[0029] The following description is made by taking a data query device based on multi-source data executing a data query method based on multi-source data as an example. Figure 1 , the data query method based on multi-source data includes:

[0030] S110: Acquire a data query request sent by the user terminal, and determine a configuration file and data query parameters according to the data query request.

[0031] Optionally, the data query device based on multi-source data provided by this solution can be designed based on the microservice architecture, using the Java language as the development language of the program or system corresponding to the data query method based on multi-source data, which can effectively ensure high performance and cross-platform compatibility, and using the Spring framework for the development and management of the corresponding program or system, using Mysql as the medium for persistent data storage. Among them, the configuration file can be recorded in json format and can be stored in a set database (such as a MySQL database). The configuration file can be stored and maintained using the set database, and the configuration file can be modified, added and deleted by modifying the database data. The data query device based on multi-source data provided by this solution can be a server machine with a JVM (Java virtual machine) installed, using virtualization technology to achieve dynamic allocation and management of resources, which can effectively ensure the stability and scalability of the system under different loads.

[0032] The configuration file provided by this solution records the request construction rules, execution call strategies and response parsing rules of one or more preset data sources. The execution call strategy can be used to record the calling strategy for one or more preset data sources. Among them, the request construction rules can be used to indicate the rules for constructing data requests for data from one or more data sources, and the corresponding execution call strategy can be constructed for the data access requirements of one or more data sources. The execution call strategy can be used to indicate the rules for obtaining data from the data source, processing data, etc., and the rules for obtaining data, processing data, etc. can be set for one or more data sources respectively. The response parsing rules can be used to indicate the processing method for the data obtained from the data source, that is, after processing, as well as the returned data type, etc. Optionally, the data source provided by this solution can be a MySQL database, a ClickHouse database (CK database), a MongoDB database, a LevelDB database, a Redis database, a key-value database, an HTTP interface, an RPC process, a custom database, etc. In a configuration file, the types corresponding to multiple data sources can be the same or different.

[0033] This solution sets the data access and processing methods of different data access interfaces and data sources through configuration files, so that data access to data sources is no longer limited to specific types of data sources, and can effectively adapt to the processing of multiple data source types and data formats, effectively expanding the applicability and flexibility in different application scenarios. Optionally, when dealing with different types of data sources and processing requirements, the strategy mode can be used to dynamically select the appropriate configuration file to achieve the selection of different processing strategies and algorithms. Each data source can correspond to a more appropriate processing strategy class to ensure the efficiency of data processing and conversion. Optionally, users can also be allowed to define processing strategies and integrate them into the configuration file for specific data processing and conversion to meet specific business needs and enhance the flexibility of the system. Some common data processing and conversion modules, such as string processing and date format conversion, can also be built into the configuration file. Users can directly call these modules in the configuration file to improve data processing efficiency. After obtaining data from multiple data sources, the processed results can be integrated and secondary processing can be performed, such as data cleaning, field mapping, aggregation calculation and other secondary processing operations to generate the final unified data output. Through the rules in the configuration file, users can define data merging strategies and result processing methods to ensure that the final output data meets business needs.

[0034] In one embodiment, when the user terminal needs to query data, it can determine the data query parameters and the data query interface corresponding to the query data based on the target object to be queried (such as a user, group, etc.), generate a data query request based on the data query parameters, and send the data query request to the data query device based on multi-source data through the above-determined data query interface. In one embodiment, the data query parameters may include the identification information of the target object (such as a platform ID and / or user ID, etc.), the region and / or country corresponding to the data, the gender and age of the target object, and one or more combinations of other customized screening rules. Among them, the target object can be a single target object or multiple target objects. Optionally, the identification information of multiple target objects can be represented by an identification information list (such as a platform ID list and / or a user ID list, etc.) that records the identification information of multiple target objects.

[0035] Exemplarily, after receiving a data query request sent by a user terminal through a data query interface, a data query device based on multi-source data parses the data query request and can determine a corresponding configuration file based on the data query interface that receives the data query request, and determine the data query parameters carried in the data query request.

[0036] Optionally, a data query device based on multi-source data may provide one or more data query interfaces for the user terminal to request data query, and different data query interfaces correspond to different configuration files. Different configuration files can be used to implement access to different data source combinations through different data query interfaces, and implement different combinations of data processing and return methods, thereby realizing flexible multi-source data query.

[0037] S120: Determine data request parameters based on the request construction rule and the data query parameters.

[0038] Exemplarily, the data request parameters are constructed according to the request construction rules in the configuration file determined above and the data query parameters corresponding to the data query request. For example, according to the required parameters and non-required parameters recorded in the request construction rules, the required parameters and non-required parameters are determined from the data query parameters and filled into the context, and the data request parameters are constructed according to the required parameters and non-required parameters in the context.

[0039] In one embodiment, the required parameters may be identification information of the target object, and the non-required parameters may be the region and / or country corresponding to the data, the gender and age of the target object, and other customized screening rules, etc. Optionally, when the data query parameters corresponding to the data query request do not include all the required parameters recorded in the request construction rule, a parameter missing reminder may be returned to the user terminal to remind the user terminal to supplement the missing required parameters.

[0040] S130: Obtaining data processing results from one or more preset data sources based on data request parameters and execution calling strategies.

[0041] Exemplarily, after obtaining the data request parameters, the data processing results are obtained from one or more preset data sources according to the data request parameters and the execution call strategy recorded in the above-determined configuration file.

[0042] For example, according to the data acquisition method for each preset data source and the processing method for the acquired data, data is acquired from each data source in turn, and the acquired data is processed to obtain the data processing results corresponding to each data source (the data processing results may include the original data acquired from the data, and / or the results of processing the original data). Optionally, the data processing results provided by this solution may be saved in the context.

[0043] S140: Perform data analysis on the data processing result based on the response analysis rule to obtain a data query result, and send the data query result to the user end.

[0044] Exemplarily, after obtaining the data processing results from one or more preset data sources, the data processing results are parsed according to the response parsing rules recorded in the configuration file to obtain the data query results, and the data query results are sent to the user end for the user end to perform subsequent processing based on the data query results. Optionally, after obtaining the data query results, the obtained data query results can be displayed in the data query interface provided to the user.

[0045] Optionally, after obtaining the data query results, they can be output to the upper-level application or user interface in a predefined format or user-configured output format. The data output can be in the form of data reports, API responses, and stored in a specified data warehouse to meet different business needs and application scenarios. Optionally, data processing results or data query results can also be processed, formatted, aggregated, etc. to improve the quality and applicability of data output, which can effectively meet complex business needs.

[0046] In one possible embodiment, the data request parameters, data processing results, and data query results provided by this solution are stored in the form of context. By saving data in the form of context, the context runs through the entire data query process, realizing the sharing of context data and the flyweight design effect of data processing. In the data query process, the data source configuration to be queried by the data query interface will be deserialized and mapped to various Java configuration entity classes in the project, and the obtained configuration class information will be placed in the context, which can effectively reduce the number of objects created during the data query process, reduce memory usage and improve performance. By sharing the same parts of multiple objects, the flyweight pattern can greatly reduce memory usage and maximize resource utilization and performance. Optionally, the configuration file can be parsed and deserialized into an object and placed in the context when the system is initialized. The configuration file related data is shared through the flyweight pattern, effectively reducing memory consumption and improving data processing efficiency.

[0047] The data source access and result integration mechanism provided by this solution can also be extended to data exchange and integration between different systems, realizing cross-system integration and data exchange. For example, the internal data integration platform or data warehouse system of an enterprise can realize data sharing and circulation between different systems through standardized interfaces and protocols, thereby improving the integration efficiency and data availability of the overall system. It can also be extended to the fields of marketing and user behavior analysis to realize the aggregation and analysis of multi-channel data. By accessing different marketing channel data (such as social media, email, advertising platforms, etc.), and adopting adaptive strategy patterns and custom processing methods, it can realize in-depth analysis of user behavior and formulation of personalized marketing strategies.

[0048] In the above, by obtaining the data query request sent by the user end, and determining the configuration file and data query parameters according to the data query request, determining the data request parameters based on the request construction rules and the data query parameters, obtaining the data processing results from one or more preset data sources based on the data request parameters and the execution call strategy, performing data analysis on the data processing results based on the response analysis rules to obtain the data query results, and sending the data query results to the user end, the request construction rules, execution call strategy and response analysis rules of the data source are defined by the configuration file, so that multiple data sources can be quickly called and configured, which can effectively reduce the cumbersome hard-coding operations of docking data sources and effectively improve the flexibility of multi-source data queries.

[0049] Based on the above embodiments, Figure 2 A flowchart of another data query method based on multi-source data provided by an embodiment of the present application is given. The data query method based on multi-source data is a specific implementation of the above-mentioned data query method based on multi-source data. Figure 2 , the data query method based on multi-source data includes:

[0050] S210: Acquire a data query request sent by the user terminal, and determine a configuration file and data query parameters according to the data query request.

[0051] S220: Determine data request parameters based on the request construction rule and the data query parameters.

[0052] S230: Obtaining data processing results from one or more preset data sources based on data request parameters and the execution calling strategy according to the execution order of the one or more data sources in the execution calling strategy.

[0053] The execution call strategy provided by this solution records the execution order of obtaining data from one or more data sources, and different data sources may correspond to different data request rules and data processing rules.

[0054] Exemplarily, this solution obtains data processing results from one or more preset data sources based on data request parameters and execution calling strategies. It can be that according to the execution order of one or more data sources in the execution calling strategy, data is obtained from the corresponding data sources based on the data request rules and data request parameters corresponding to the data sources, and the data obtained from the data source is processed based on the data processing rules corresponding to the data source to obtain the data processing results.

[0055] Optionally, the execution order of each data source can be determined by an execution order identifier (such as order). For example, when the data source includes a ClickHouse database and two different http2brpc databases, at this time, the execution order identifier order of the ClickHouse database is 1, the execution order identifier order of the first http2brpc database is 2, and the execution order identifier order of the second http2brpc database is 3, then the execution order is to execute the data acquisition and processing corresponding to the ClickHouse database, the first http2brpc database, and the second http2brpc database in sequence. By setting the execution order of different data sources, this solution can start the corresponding processors in sequence to process when executing data acquisition and processing of different data sources according to the execution order, forming a responsibility chain mode for data call and processing, which can better organize and process requests of multiple data source interfaces, and realize dynamic processing and response of data query requests. In addition, the rules in the configuration file can be parsed into a series of processors, and a responsibility chain is formed according to the execution order, and each processor is responsible for a specific data source operation. When a data query request arrives at the system (a data query device based on multi-source data), it passes through each processor in the responsibility chain in turn, dynamically acquires and processes data according to the configuration rules, thereby achieving flexible data source access, automatic routing and processing of data query requests, and effectively optimizing data access and processing processes.

[0056] In one possible embodiment, Figure 3 As shown in a data processing result acquisition process diagram provided, the data query method based on multi-source data provided by the present solution obtains data processing results from one or more preset data sources based on data request parameters and execution call strategies, including:

[0057] S231: constructing a data request instruction based on the data request parameters and the request parameter rules for the data source in the execution call strategy, and obtaining a data request result from the data source according to the data request instruction.

[0058] S232: Process the data request result according to the result processing rule for the data source in the execution call strategy to obtain the data processing result of the data source.

[0059] The execution call strategy provided by this solution records the execution order, request parameter rules and result processing rules corresponding to each data source, where the request parameter rules can be used to indicate the rules for constructing data request instructions sent to the corresponding data source, and the result processing rules are used to indicate the processing rules for data obtained from the data source. Different data sources can correspond to different request parameter rules and result processing rules, and the request parameter rules and result processing rules can be set based on the data access requirements, stored data types, data characteristics, and data processing requirements corresponding to different data sources.

[0060] Exemplarily, data is acquired and processed from each data source in sequence according to the execution order preset in the execution call strategy. For each data source, according to the data request parameters determined above and the request parameter rules for the corresponding data source in the execution call strategy recorded in the configuration file, a data request instruction for requesting data from the data source is constructed, and data is requested from the corresponding data source according to the data request instruction, so as to obtain the data request result from the data source.

[0061] In one embodiment, for each data source, after obtaining the data request result from the data source, the data request result obtained from the corresponding data source is processed according to the result processing rule for the data source in the execution call strategy recorded in the configuration file to obtain the data processing result of the data source responding to the data query request. This solution constructs a data request instruction according to data request parameters and request parameter rules, obtains data request results from each data source according to the data request instruction, processes the obtained data request results according to the result processing rules to obtain the data processing results of each data source, accurately obtains data request results from each data source, and processes the data request results of each data source to obtain the data processing results, obtains and processes data for different data sources, can flexibly adapt to the access requirements of different data sources, improves the flexibility of data query, and can also configure different request parameter rules and result processing rules for different data sources to improve the scalability of different data sources.

[0062] In a possible embodiment, the data query method based on multi-source data provided by the present solution processes the data request result according to the result processing rule of the data source in the execution call strategy to obtain the data processing result of the data source, including:

[0063] S2321: Process the data request result according to the result processing rule for the data source in the execution call strategy to obtain a first data processing result of the data source.

[0064] S2322: Perform data processing on the first data processing result according to the post-processing rule for the data source in the execution call strategy to obtain a second data processing result of the data source.

[0065] In one embodiment, the execution call strategy provided by the present solution may also include post-processing rules, which may be used to indicate processing rules for the first data processing result obtained by processing data based on the result processing rules. Optionally, result processing rules and post-processing rules for different data sources may be set according to the data processing requirements of different data sources, and appropriate processing rules may be dynamically selected according to different data sources and processing requirements to ensure efficient processing and conversion of data and improve system performance and scalability.

[0066] Exemplarily, after obtaining the data request result from the data source, the data request result is processed according to the result processing rule of the data source in the execution call strategy to obtain the first data processing result of the data source responding to the data query request. In one embodiment, after obtaining the first data processing result, according to the post-processing rule of the data source in the execution call strategy, the first data processing result is processed to obtain the second data processing result of the data source responding to the data query request. At this time, the data processing result of the data source includes the first data processing result and the second data processing result determined above.

[0067] For example, after obtaining the data request result recorded in the form of a key-value pair from the data source ClickHouse database, the obtained data request result is converted into data in Map format as the first data processing result according to the result processing rule, and then the first data processing result in Map format is converted into data in table (List) format as the second data processing result according to the post-processing rule. This solution effectively enriches the data processing methods for different data sources by performing data processing on the data request result based on the result processing rule to obtain the first data processing result, and performing data processing on the first data processing result based on the post-processing rule to obtain the second data processing result. It can also write result processing rules and post-processing rules according to business needs, enhance the customization ability and adaptability of the system, and enhance the customized processing ability and adaptability of multiple data sources.

[0068] In one embodiment, the data query method based on multi-source data provided by the present solution obtains data processing results from one or more preset data sources based on data request parameters and an execution call strategy, which may be: when the execution call strategy does not indicate the use of custom rules, the data processing results are obtained from one or more preset data sources based on the data request parameters and the execution call strategy; when the execution call strategy indicates the use of custom rules, the data processing results are obtained from one or more preset data sources based on the data request results and the custom rules.

[0069] Optionally, general result processing rules and / or post-processing rules may be configured for each data source or a combination of different data sources. Customized rules may also be configured for each data source or a combination of different data sources. Whether to use custom rules may be configured in the execution call strategy of the configuration file.

[0070] Exemplarily, after determining the data request parameters based on the request construction rule and the data query parameters, determine whether the execution call policy of the configuration file indicates the use of custom rules. For example, a preset policy identifier (e.g., executeCustomMethod) is used to indicate whether to use custom rules. When the value corresponding to the policy identifier is a first identifier (e.g., executeCustomMethod=true), it is considered that the execution call policy does not indicate the use of custom rules, and when the value corresponding to the policy identifier is a second identifier (e.g., executeCustomMethod=false), it is considered that the execution call policy indicates the use of custom rules.

[0071] When the execution call policy does not indicate the use of custom rules, the data processing results are obtained from one or more preset data sources based on the data request parameters and the execution call policy. When the execution call policy indicates the use of custom rules, the data processing results are obtained from one or more preset data sources based on the data request results and the custom rules, and there is no need to process the data according to the general result processing rules and / or post-processing rules in the execution call policy. This solution obtains data processing results from the data source based on the execution call policy when the execution call policy does not indicate the use of custom rules, and obtains data processing results from the data source based on the custom rules when the execution call policy indicates the use of custom rules. This meets different data acquisition and processing requirements under multiple data sources, achieves flexible response to different data sources and processing requirements, ensures efficient processing and conversion of data, and can more accurately adapt to complex business scenarios and data processing requirements.

[0072] S240: Perform data analysis on the data processing result based on the response analysis rule to obtain a data query result, and send the data query result to the user end.

[0073] In one embodiment, the data query method based on multi-source data provided by the present solution performs data analysis on the data processing results based on the response analysis rules to obtain the data query results. The data query results can be obtained by performing data analysis on the data processing results based on one or more combinations of the data processing rules for the data source in the response analysis rules, the field information directly returned, and the field information subjected to secondary processing.

[0074] The response parsing rules provided by this solution record one or more combinations of data processing rules for each data source, field information directly returned for secondary processing, etc. Among them, the data processing method can be understood as the processing method of the data processing results, such as merging the data processing results, data type conversion processing, etc., the field information directly returned can be understood as the fields in the data processing results that can be directly returned to the user end, and the field information for secondary processing can be understood as the fields that need to be secondary processed. For example, secondary processing can be to assign values ​​to corresponding fields (for example, assign values ​​to corresponding fields based on values ​​determined by preset expressions or based on values ​​determined by custom functions), obtain data based on website information in the data processing results (for example, obtain the corresponding image based on the image URL address), and convert data (for example, time format conversion, converting the time in timestamp format to time in a common time format (for example, year-month-day-hour-minute-second format)).

[0075] Exemplarily, after obtaining the data processing results corresponding to one or more data sources, determine whether the response parsing rules record the data processing rules for the data source, the field information directly returned, and the field information for secondary processing, and perform data parsing on the data processing results according to the data processing rules for the data source, the field information directly returned, and the field information for secondary processing in the response parsing rules to obtain the data query results. This solution flexibly configures the data parsing processing rules for the data processing results by configuring one or more combinations of data processing rules, field information directly returned, and field information for secondary processing in the parsing rules, which can effectively meet different business needs and application scenarios.

[0076] In one embodiment, when acquiring and processing data from different data sources, a plug-in architecture and / or a distributed task scheduling framework may be used to acquire and process data sources, wherein the plug-in architecture can more flexibly expand and manage various data sources, and the distributed task scheduling framework can achieve more complex task scheduling and resource management. In addition, a rule engine and / or a process orchestration tool may be used to define and execute data processing processes, wherein the rule engine may perform complex logical processing on data based on a rule set, and the process orchestration tool may effectively manage the execution order and dependencies of the entire data processing process.

[0077] In one embodiment, the solution can define the connection information and data extraction method of the data source through the configuration file, realize the rapid docking and configuration of multiple data sources, avoid cumbersome hard-coding operations, effectively improve the flexibility and scalability of the system, and realize flexible docking of multiple data sources. At the same time, the configuration file is easy to understand and modify, and non-professionals can also configure and maintain the system, effectively reducing the system maintenance cost, and the data source and display rules can be quickly adjusted through the configuration file, which can quickly respond to changes in business needs, reduce the time of development, testing and deployment, improve the response speed of the system, and enhance the business agility of the enterprise. The data processing and conversion module can also be built into the configuration file to support operations such as cleaning, conversion and merging of the extracted data, provide a high-quality data foundation for data display, define the display rules through the configuration file, realize flexible display of data, simplify the secondary processing process of data display, and improve data processing efficiency.

[0078] In the above, by obtaining the data query request sent by the user end, and determining the configuration file and data query parameters according to the data query request, determining the data request parameters based on the request construction rules and the data query parameters, obtaining the data processing results from one or more preset data sources based on the data request parameters and the execution call strategy, performing data analysis on the data processing results based on the response analysis rules to obtain the data query results, and sending the data query results to the user end, the request construction rules, execution call strategy and response analysis rules of the data source are defined in the configuration file, so that the rapid call and configuration of multiple data sources can be realized, which can effectively reduce the cumbersome hard-coding operations of docking data sources and effectively improve the flexibility of multi-source data query. At the same time, by setting the execution order of different data sources, when executing data acquisition and processing of different data sources according to the execution order, the corresponding processors can be started in sequence for processing, forming a data call and processing responsibility chain mode, which can better organize and process the requests of multiple data source interfaces and realize the dynamic processing and response of data query requests. Furthermore, by recording in the response parsing rules the data processing rules for each data source, the field information directly returned for secondary processing, or one or more combinations of these, a unified data processing flow and result integration mechanism can be formed to ensure data consistency and accuracy and reduce data inconsistencies or errors caused by access to multiple data sources.

[0079] Figure 4 Schematic diagram of a data query device based on multi-source data provided by an embodiment of the present application. Figure 4 The data query device based on multi-source data includes a request response module 41, a parameter determination module 42, a data acquisition module 43 and a result determination module 44.

[0080] Among them, the request response module 41 is configured to obtain the data query request sent by the user end, and determine the configuration file and data query parameters according to the data query request, the configuration file records the request construction rules, execution call strategy and response parsing rules of one or more preset data sources, and the execution call strategy is used to record the calling strategy for one or more preset data sources; the parameter determination module 42 is configured to determine the data request parameters based on the request construction rules and the data query parameters; the data acquisition module 43 is configured to obtain the data processing results from one or more preset data sources based on the data request parameters and the execution call strategy; the result determination module 44 is configured to perform data analysis on the data processing results based on the response parsing rules to obtain the data query results, and send the data query results to the user end.

[0081] In the above, by obtaining the data query request sent by the user end, and determining the configuration file and data query parameters according to the data query request, determining the data request parameters based on the request construction rules and the data query parameters, obtaining the data processing results from one or more preset data sources based on the data request parameters and the execution call strategy, performing data analysis on the data processing results based on the response analysis rules to obtain the data query results, and sending the data query results to the user end, the request construction rules, execution call strategy and response analysis rules of the data source are defined by the configuration file, so that multiple data sources can be quickly called and configured, which can effectively reduce the cumbersome hard-coding operations of docking data sources and effectively improve the flexibility of multi-source data queries.

[0082] In one possible embodiment, the data acquisition module 43 obtains data processing results from one or more preset data sources based on data request parameters and execution call strategies, and is configured to: obtain data processing results from one or more preset data sources based on data request parameters and execution call strategies in accordance with the execution order of one or more data sources in the execution call strategies.

[0083] In a possible embodiment, the data acquisition module 43 acquires data processing results from one or more preset data sources based on data request parameters and execution call strategies, and is configured as follows:

[0084] Building a data request instruction based on the data request parameters and the request parameter rules for the data source in the execution call strategy, and obtaining the data request result from the data source according to the data request instruction;

[0085] The data request result is processed according to the result processing rule for the data source in the execution call strategy to obtain the data processing result of the data source.

[0086] In a possible embodiment, the data acquisition module 43 performs data processing on the data request result according to the result processing rule for the data source in the execution call strategy to obtain the data processing result of the data source, and is configured as follows:

[0087] Perform data processing on the data request result according to the result processing rule for the data source in the execution call strategy to obtain a first data processing result of the data source;

[0088] The first data processing result is processed according to the post-processing rule for the data source in the execution calling strategy to obtain a second data processing result of the data source.

[0089] In a possible embodiment, the data acquisition module 43 acquires data processing results from one or more preset data sources based on data request parameters and execution call strategies, and is configured as follows:

[0090] When the execution call strategy does not indicate the use of custom rules, obtain data processing results from one or more preset data sources based on data request parameters and the execution call strategy;

[0091] When the execution call strategy indicates the use of custom rules, the data processing results are obtained from one or more preset data sources based on the data request results and the custom rules.

[0092] In a possible embodiment, the result determination module 44 performs data analysis on the data processing result based on the response analysis rule to obtain the data query result, and is configured as: based on one or more combinations of the data processing rule for the data source in the response analysis rule, the field information directly returned, and the field information subjected to secondary processing, the data processing result is analyzed to obtain the data query result.

[0093] In a possible embodiment, data request parameters, data processing results, and data query results are stored in the form of context.

[0094] It is worth noting that in the above-mentioned embodiment of the data query device based on multi-source data, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present application.

[0095] The embodiment of the present application also provides a data query device based on multi-source data, and the data query device based on multi-source data can be integrated with the data query apparatus based on multi-source data provided in the embodiment of the present application. Figure 5 Schematic diagram of a data query device based on multi-source data provided by an embodiment of the present application. Figure 5 The data query device based on multi-source data includes: an input device 53, an output device 54, a memory 52 and one or more processors 51; the memory 52 is used to store one or more programs; when one or more programs are executed by one or more processors 51, the one or more processors 51 implement the data query method based on multi-source data provided in the above embodiment. The data query device, equipment and computer based on multi-source data provided above can be used to execute the data query method based on multi-source data provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0096] The embodiments of the present application also provide a non-volatile storage medium storing computer executable instructions, which are used to execute the data query method based on multi-source data provided in the above embodiments when executed by a computer processor. Of course, the non-volatile storage medium storing computer executable instructions provided in the embodiments of the present application, whose computer executable instructions are not limited to the data query method based on multi-source data provided above, can also execute the related operations in the data query method based on multi-source data provided in any embodiment of the present application. The data query device, equipment and storage medium based on multi-source data provided in the above embodiments can execute the data query method based on multi-source data provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the data query method based on multi-source data provided in any embodiment of the present application.

[0097] On the basis of the above embodiments, the embodiments of the present application also provide a computer program product. The technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer program product is stored in a storage medium, including a number of instructions for enabling a computer device, a mobile terminal or a processor therein to execute all or part of the steps of the data query method based on multi-source data provided in each embodiment of the present application.

Claims

1. A data query method based on multi-source data, It is characterized in that include: Obtaining a data query request sent by a user terminal, and determining a configuration file and data query parameters according to the data query request, wherein the configuration file records a request construction rule, an execution calling strategy, and a response parsing rule of one or more preset data sources, and the execution calling strategy is used to record a calling strategy for one or more preset data sources; Determine data request parameters based on the request construction rule and the data query parameters; Obtaining data processing results from one or more preset data sources based on the data request parameters and the execution call strategy; The data processing result is subjected to data analysis processing based on the response analysis rule to obtain a data query result, and the data query result is sent to the user terminal.

2. The data query method based on multi-source data according to claim 1, It is characterized in that The obtaining of data processing results from one or more preset data sources based on the data request parameters and the execution call strategy includes: According to the execution order of one or more data sources in the execution calling strategy, data processing results are obtained from one or more preset data sources based on the data request parameters and the execution calling strategy.

3. The data query method based on multi-source data according to claim 1, It is characterized in that The obtaining of data processing results from one or more preset data sources based on the data request parameters and the execution call strategy includes: Building a data request instruction based on the data request parameter and the request parameter rule for the data source in the execution call strategy, and obtaining a data request result from the data source according to the data request instruction; The data request result is processed according to the result processing rule for the data source in the execution call strategy to obtain the data processing result of the data source.

4. The data query method based on multi-source data according to claim 3, It is characterized in that The performing data processing on the data request result according to the result processing rule for the data source in the execution call strategy to obtain the data processing result of the data source includes: Performing data processing on the data request result according to the result processing rule for the data source in the execution call strategy to obtain a first data processing result of the data source; The first data processing result is processed according to the post-processing rule for the data source in the execution calling strategy to obtain a second data processing result of the data source.

5. The data query method based on multi-source data according to claim 1, It is characterized in that The obtaining of data processing results from one or more preset data sources based on the data request parameters and the execution call strategy includes: When the execution calling strategy does not indicate the use of a custom rule, obtaining a data processing result from one or more preset data sources based on the data request parameter and the execution calling strategy; In the case where the execution call strategy indicates the use of custom rules, a data processing result is obtained from one or more preset data sources based on the data request result and the custom rules.

6. The data query method based on multi-source data according to claim 1, It is characterized in that The performing data analysis on the data processing result based on the response analysis rule to obtain the data query result includes: Based on one or more combinations of the data processing rules for the data source in the response parsing rules, the directly returned field information, and the field information for secondary processing, the data processing result is parsed to obtain a data query result.

7. The data query method based on multi-source data according to claim 1, It is characterized in that The data request parameters, the data processing results and the data query results are stored in the form of context.

8. A data query device based on multi-source data, It is characterized in that It includes a request response module, a parameter determination module, a data acquisition module and a result determination module, wherein: The request response module is configured to obtain a data query request sent by a user terminal, and determine a configuration file and data query parameters according to the data query request, wherein the configuration file records a request construction rule, an execution call strategy, and a response parsing rule of one or more preset data sources, and the execution call strategy is used to record a call strategy for one or more preset data sources; The parameter determination module is configured to determine the data request parameters based on the request construction rule and the data query parameters; The data acquisition module is configured to acquire data processing results from one or more preset data sources based on the data request parameters and the execution call strategy; The result determination module is configured to perform data analysis processing on the data processing result based on the response analysis rule to obtain a data query result, and send the data query result to the user terminal.

9. A data query device based on multi-source data, It is characterized in that include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data query method based on multi-source data as described in any one of claims 1 to 7.

10. A non-volatile storage medium storing computer executable instructions, It is characterized in that When the computer executable instructions are executed by a computer processor, they are used to execute the data query method based on multi-source data as described in any one of claims 1 to 7.

11. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the data query method based on multi-source data described in any one of claims 1 to 7 is implemented.

Images