Data processing method, device, storage medium and electronic device

Abstract

The present disclosure provides a data processing method, device, storage medium and electronic device, which relate to the field of computer technology. The method comprises: configuring a rule script and a query template through a preset service interface; when a message subscription request is received through the preset service interface, obtaining the original message data corresponding to the message subscription request; determining the rule script corresponding to the original message data, and extracting the target field in the original message data according to the rule script; combining the query template and the target field, generating a target query statement; using the target query statement to query the database, and obtaining the associated data of the original message data. In the present disclosure, by configuring the rule script and the query template, the associated query of the message data is simplified, and the data query efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a data processing method, a data processing device, a computer-readable storage medium, and an electronic device. Background Art

[0002] With the rapid development of computer technology, users have higher and higher requirements for data synchronization. In the message subscription push scenario, the message data can be pushed from the data source to the subscription system to achieve message data synchronization. In certain specific scenarios, the message data needs to contain some related data. For example, for a piece of user data, in addition to the user's data information, it can also contain related data such as the user's associated role, project, organization, etc., to facilitate subsequent data search and data filtering in different dimensions.

[0003] At present, since different types of related data may cause each related data to exist in different types of databases, when querying related data, it is necessary to call different interfaces in different ways to query data, making the entire data query process more complicated, thereby reducing data query efficiency.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a data processing method, a data processing device, a computer-readable storage medium and an electronic device, thereby overcoming, at least to a certain extent, the problem in the related art that the process of associated query of message data is relatively complicated, resulting in low data query efficiency.

[0006] According to a first aspect of the present disclosure, there is provided a data processing method, comprising:

[0007] Configure rule scripts and query templates through preset service interfaces;

[0008] When a message subscription request is received through the preset service interface, obtaining original message data corresponding to the message subscription request;

[0009] Determine a rule script corresponding to the original message data, and extract a target field in the original message data according to the rule script;

[0010] Combining the query template and the target field to generate a target query statement;

[0011] The target query statement is used to perform a database query to obtain associated data of the original message data.

[0012] In an exemplary embodiment of the present disclosure, before obtaining the original message data corresponding to the message subscription request, the method further includes:

[0013] A data source is configured through the preset service interface, and the data source includes at least one of a message queue and a database log.

[0014] In an exemplary embodiment of the present disclosure, when a message subscription request is received through the preset service interface, obtaining original message data corresponding to the message subscription request includes:

[0015] When a message subscription request is received through the preset service interface, determining a target data source according to a message type in the message subscription request;

[0016] The original message data corresponding to the message subscription request is obtained from the target data source.

[0017] In an exemplary embodiment of the present disclosure, determining a rule script corresponding to the original message data, and extracting a target field in the original message data according to the rule script includes:

[0018] Determine a rule script corresponding to the original message data according to the message type of the original message data;

[0019] The rule script is executed by a script engine to extract the target field from the original message data.

[0020] In an exemplary embodiment of the present disclosure, the database includes at least one of a relational database and a non-relational database; the step of querying the database using the target query statement to obtain associated data of the original message data includes:

[0021] Using a query engine to convert the target query statement into a query plan for the database;

[0022] The database is queried based on the query plan to obtain associated data of the original message data.

[0023] In an exemplary embodiment of the present disclosure, after obtaining the associated data of the original message data, the method further includes:

[0024] The associated data of the original message data is cached, and a mapping relationship between the associated data and the target field is configured.

[0025] In an exemplary embodiment of the present disclosure, after obtaining the associated data of the original message data, the method further includes:

[0026] combining the original message data and the associated data of the original message data to obtain target message data;

[0027] The target message data is pushed to the corresponding message subscription system.

[0028] In an exemplary embodiment of the present disclosure, the query template is a SQL template.

[0029] According to a second aspect of the present disclosure, there is provided a data processing device, comprising:

[0030] The associated query configuration module is used to configure rule scripts and query templates through the preset service interface;

[0031] A message data acquisition module, configured to acquire original message data corresponding to a message subscription request when a message subscription request is received through the preset service interface;

[0032] A query condition extraction module, used to determine the rule script corresponding to the original message data, and extract the target field in the original message data according to the rule script;

[0033] A query statement generating module, used to generate a target query statement by combining the query template and the target field;

[0034] The associated data query module is used to perform a database query using the target query statement to obtain the associated data of the original message data.

[0035] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the methods described above is implemented.

[0036] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0037] The exemplary embodiments of the present disclosure may have some or all of the following beneficial effects:

[0038] In the data processing method provided in the exemplary implementation of the present disclosure, a rule script and a query template are configured through a preset service interface; when a message subscription request is received through the preset service interface, the original message data corresponding to the message subscription request is obtained; the rule script corresponding to the original message data is determined, and the target field in the original message data is extracted according to the rule script; a target query statement is generated by combining the query template and the target field; the target query statement is used to query the database to obtain the associated data of the original message data. In the present disclosure, by configuring the rule script and the query template, the associated query of the message data is simplified and the data query efficiency is improved.

[0039] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0041] Figure 1 A schematic diagram showing an exemplary system architecture to which the data processing method and device according to the embodiments of the present disclosure can be applied;

[0042] Figure 2 A flowchart of a data processing method in an embodiment of the present disclosure is schematically shown;

[0043] Figure 3 A flowchart for extracting query conditions in an embodiment of the present disclosure is schematically shown;

[0044] Figure 4 A flowchart of a database query in an embodiment of the present disclosure is schematically shown;

[0045] Figure 5 A flowchart schematically shows another data processing method in an embodiment of the present disclosure;

[0046] Figure 6 A block diagram of a data processing device in an embodiment of the present disclosure is schematically shown;

[0047] Figure 7 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0048] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0049] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0050] Figure 1 A schematic diagram of a system architecture of an exemplary application environment to which a data processing method and device according to an embodiment of the present disclosure can be applied is shown.

[0051] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104 and a server 105. The terminal devices 101, 102, 103 may be various electronic devices, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. The server 105 may be a single server, or a server cluster consisting of multiple servers, or a virtualization platform or a cloud computing service center. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0052] The data processing method provided in the exemplary embodiment of the present disclosure can be executed by the server 105, and accordingly, the data processing device can be set in the server 105. For example, after the server 105 receives the message subscription request input by the terminal device 101, by executing the data processing method, the associated data obtained by query can be sent to the terminal device 101 to be displayed to the user. However, it is easy for those skilled in the art to understand that the data processing method provided in the exemplary embodiment of the present disclosure can also be executed by one or more of the terminal devices 101, 102, and 103, and accordingly, the data processing device can also be set in the terminal devices 101, 102, and 103. For example, the terminal device 101 executes the data processing method, and the associated data obtained by query can be directly displayed on the display screen of the terminal device 101 to be displayed to the user, and the present disclosure does not make any special limitation on this.

[0053] The technical solution of the embodiment of the present disclosure is described in detail below:

[0054] With the rapid development of computer technology, users have higher and higher requirements for data synchronization. For example, different business scenarios require different database capabilities, in which heterogeneous databases synchronize data based on messages of change events. For example, for user data in a MySQL (relational database) user data change event, if the user data only contains the user's data information, but does not contain related data such as the user's associated role, project, organization, etc., when the user data is synchronized to the search engine, it is impossible to search the user data in different dimensions, and when the user data is synchronized to an external system, it is impossible to filter the user data in other dimensions.

[0055] At present, since different types of related data may cause each related data to exist in different types of databases, when querying related data, it is necessary to call different interfaces in different ways to query data, making the entire data query process more complicated, thereby reducing data query efficiency.

[0056] Based on this problem, this example implementation provides a data processing method, which simplifies the associated query of message data by configuring rule scripts and SQL (Structured Query Language) templates, thereby improving data query efficiency. Figure 2 As shown, the data processing method may include the following steps S210 to S250:

[0057] Step S210. Configure the rule script and query template through the preset service interface;

[0058] Step S220. When a message subscription request is received through the preset service interface, the original message data corresponding to the message subscription request is obtained;

[0059] Step S230. Determine the rule script corresponding to the original message data, and extract the target field in the original message data according to the rule script;

[0060] Step S240. Combining the query template and the target field, generating a target query statement;

[0061] Step S250: Use the target query statement to perform a database query to obtain associated data of the original message data.

[0062] In the data processing method provided in the exemplary implementation of the present disclosure, a rule script and a query template are configured through a preset service interface; when a message subscription request is received through the preset service interface, the original message data corresponding to the message subscription request is obtained; the rule script corresponding to the original message data is determined, and the target field in the original message data is extracted according to the rule script; a target query statement is generated by combining the query template and the target field; the target query statement is used to query the database to obtain the associated data of the original message data. In the present disclosure, by configuring the rule script and the query template, the associated query of the message data is simplified and the data query efficiency is improved.

[0063] Next, the above steps of this exemplary embodiment are described in more detail.

[0064] In step S210, the rule script and query template are configured through a preset service interface.

[0065] In the message subscription push scenario, the preset service interface may be a message subscription association processing service interface. Correspondingly, the user may configure the rule script and query template through the message subscription association processing service interface. The rule script is a script code used to extract data from the message data. Different rule scripts may be configured for different subscription events. The example implementation of the present disclosure may support multiple scripting languages, such as javascript scripts, groovy scripts, etc. The query template is an SQL template. The SQL template may include an SQL query statement. The SQL query statement in the SQL template may be modified later to dynamically generate a new SQL query statement, so as to use the new SQL query statement to perform database query, thereby improving the flexibility of data query.

[0066] For example, a rule script for a subscription event configuration can be:

[0067] "extractshell":"if(json.data.projectcode!=null)\n[json.data.projectcode]\nelse\n[]\n"

[0068] This rule script can be used to extract projectcode (project information) from the original message data.

[0069] The configured SQL template can be:

[0070] "sqlTemplate": "select*from project where name=?"

[0071] The corresponding data in the database can be queried through the SQL query statement in the SQL template.

[0072] In the example implementation of the present disclosure, the data source can also be configured through the message subscription association processing service interface, wherein the data source can include at least one of a message queue and a database log. For example, the data source can be a kafka message queue, etc., or a MySQL binlog (binary) log event, etc. For example, a configured data source format can include at least a message data identifier, an event type, a message data description, a message subject, and other data, and the other data can be a kafka cluster, a data format, etc., which is not specifically limited in the present disclosure. It should be noted that the service address, message filtering conditions, and cache information of the message subscription system can also be configured through the message subscription association processing service interface. In addition, after creating a subscription, the message subscription association processing service interface can be called to bind the subscription to the message type, and when there is a new message event, the message event will be pushed to the subscription.

[0073] In this example, by configuring the rule script and SQL template, it is convenient to implement the associated query of message data later without learning and writing complex message processing code, thus simplifying the associated query of message data.

[0074] In step S220, when a message subscription request is received through the preset service interface, original message data corresponding to the message subscription request is obtained.

[0075] In the example implementation of the present disclosure, a user may send a message subscription request through a message subscription association processing service interface, and the message subscription request includes data identification, message type, etc. When a message subscription request is received, a target data source may be determined according to the message type in the message subscription request to obtain the original message data corresponding to the message subscription request from the target data source.

[0076] For example, a specific data format for sending a message subscription request may be:

[0077] POST{{baseUrl}} / admin / event / type

[0078] Content-Type:application / json

[0079] userid:{{uesrID}}

[0080] projected:0

[0081] When the message subscription request is received, the original message data can be pulled from the target data source according to the message type and data identifier, etc. In this example, the reading of Kafka message queues and database log change data is supported.

[0082] In step S230, the rule script corresponding to the original message data is determined, and the target field in the original message data is extracted according to the rule script.

[0083] After obtaining the original message data, the corresponding rule script can be run through the script engine according to the configuration to extract the target field in the original message data as the SQL query condition, and then generate a complete SQL query statement.

[0084] In one example implementation, reference is made to Figure 3 As shown, query conditions can be extracted from the original message data according to step S301 and step S302 to facilitate subsequent generation of SQL query statements.

[0085] In step S301, a rule script corresponding to the original message data is determined according to the message type of the original message data.

[0086] It is understandable that different subscription events are configured with different rule scripts, and subscription is bound to the message type. Therefore, the rule script corresponding to the original message data can be determined according to the message type of the original message data.

[0087] In step S302, the rule script is executed by using a script engine to extract the target field from the original message data.

[0088] The message subscription association processing service can use the script engine to execute the rule script to extract the target field from the original message data, such as the target field can be a user ID, etc. For example, for a user table update data read from MySQL binlog, when it is necessary to supplement the organization information and project information associated with the user, the user ID value: 1498 and the organization ID value: a23c4 in the update data can be extracted by executing the rule script, and the user ID value and the organization ID value can be used as SQL query conditions.

[0089] In this example, the message subscription association processing service is combined with the script engine. By configuring rule scripts and SQL templates, the associated data of message events can be completed, simplifying the associated query of message data.

[0090] In step S240, a target query statement is generated by combining the query template and the target field.

[0091] In the example implementation of the present disclosure, the target query statement may be an SQL query statement. Exemplarily, after the target field is extracted from the original message data, the extracted target field may be used as a query condition, and a complete SQL query statement is generated by combining the SQL template and the target field. For example, when the extracted target field is the user id value: 1498 and the organization id value: a23c4, the user id value and the organization id value may be used as SQL query conditions, and the SQL template may be modified according to the SQL query conditions to generate two complete SQL query statements, such as:

[0092] 1)select name,displayName from USERCENTERDB.project as t1 joinUSERCENTERDB.projectUser as t2 on t1.id=t2.projectId where t2.userId=1498;

[0093] 2)select id,name,parent_ids from USERCENTERDB.company where id="a23c".

[0094] Among them, SQL query statement 1) is used to query project information from the database, and SQL query statement 2) is used to query organization information from the database.

[0095] In this example, based on the pre-configured SQL template, SQL query statements can be generated flexibly and efficiently, which facilitates the associated query of message data and thus improves data query efficiency.

[0096] In step S250, the target query statement is used to perform a database query to obtain associated data of the original message data.

[0097] After generating a complete SQL query statement, the SQL query statement can be used to query the database, wherein the database can be at least one of a relational database and a non-relational database (also known as a Nosql database). Relational databases support SQL queries, while Nosql databases do not support SQL queries. NoSql databases can be ElasticSearch databases, Mongodb databases, redis databases, etc., which are not limited in this disclosure.

[0098] In one example implementation, reference is made to Figure 4 As shown, the associated data query can be performed according to step S401 and step S402. Among them, the SQL syntax can be uniformly used to support data query of various databases (relational databases and NoSql databases).

[0099] In step S401, the target query statement is converted into a query plan of the database using a query engine.

[0100] It is understandable that the associated data of the original message data may exist in databases of different architectures, such as in a relational database that supports SQL queries and a NoSql type database that does not support SQL queries. Exemplarily, a complete SQL query statement can be input into a query module of a message subscription association processing service, and the query module can use the Apache Calcite engine to convert the SQL query statement into a query plan for various databases, so as to query the corresponding database through each query plan. Among them, the Apache Calcite engine is a complete query processing system that can provide many common functions in addition to data storage and management, such as query execution, optimization, and query language.

[0101] In step S402, the database is queried based on the query plan to obtain associated data of the original message data.

[0102] Then, you can request the database to query data by executing the query plan of various databases to obtain the associated data of the original message data. Among them, the query plan is also called the execution plan or the explanation plan, which is the specific steps of the database to execute the SQL query statement. For example, the query plan refers to the access to the data in the table through the index or the full table scan, the implementation method of the connection query and the order of the connection, etc.

[0103] For example, after converting SQL query statement 1) and SQL query statement 2) into query plans of the corresponding databases and performing database queries, the corresponding associated data of the original message data obtained by the query can be the user's project information and organization information. The project information can include project id values, project names, etc., and the organization information can include organization id values, organization names, and other information.

[0104] In this example, SQL syntax is uniformly used to query related data for relational databases and Nosql databases. There is no need to call multiple interfaces to query different databases, which improves data query efficiency.

[0105] After obtaining the associated data of the original message data, the original message data and the associated data of the original message data can be combined to obtain the target message data, and the target message data can be pushed to the corresponding message subscription system. For example, the push of message data can be achieved through HTTP (HyperText Transfer Protocol) protocol, websocket (a network protocol based on TCP) protocol, etc. Among them, the message subscription system can be a subscription service or a database, that is, the target message data is synchronized to the database, and the database can be a mongodb database, elasticSearch database, etc. Exemplarily, after the associated data is supplemented to the original message data, it can be filtered and pushed according to the subscription conditions to determine whether to distribute the complete message data to the target subscription service or synchronize the complete message data to the target database.

[0106] It is understandable that since the original message data is completed using associated data, correspondingly, a wider range of filtering processing can be implemented later. For example, the completed original message data includes not only user data, but also organization information, project information, etc. For example, user information belonging to a certain organization can be pushed to the target message subscription system based on the organization information. If the target message subscription system is a database that needs to be synchronized, such as an ElasticSearch database, after synchronizing the completed original message data to the ElasticSearch database, a document object can be created based on the completed original message data. The document object can include indexes of different dimensions, such as project dimensions, organization dimensions, etc., and is no longer limited to user information dimensions.

[0107] After obtaining the associated data of the original message data, you can also cache the associated data of the original message data and configure the mapping relationship between the associated data and the target field. After extracting the target field from the original message data, you can use the target field as a query condition. For message data containing the same query condition, you can directly obtain the corresponding associated data from the cache. For example, based on the mapping relationship between the associated data and the target field, you can query using the target field to obtain the associated data corresponding to the target field without having to perform SQL queries, thereby increasing the speed of message data distribution.

[0108] In this example, the query and completion of message event-related data are performed by configuring rule scripts and SQL templates, which simplifies the associated query of message data and improves data query efficiency. Moreover, SQL syntax is uniformly used to query associated data for relational databases and Nosql databases, without calling multiple interfaces to query different databases, further improving data query efficiency. In addition, compared to big data frameworks such as Spark and flink, the present disclosure proposes a lightweight data processing method that does not require learning and writing complex data processing codes. Data association query, filtering and other processing can be achieved by submitting relevant configurations and SQL templates.

[0109] In one example implementation, reference is made to Figure 5 As shown, the associated query of the message data can be implemented according to steps S501 to S505.

[0110] Step S501. User submits subscription and associated query configuration: The user configures the data source, message subscription push system service address, message filtering conditions, associated rule scripts, SQL templates and cache information, etc. by calling the message subscription associated processing service interface;

[0111] Step S502. Obtaining original message data: When there is a new message event, extracting the original message data from the target data source, wherein the target data source may be a message queue system, or a database log event, etc.;

[0112] Step S503. Generate SQL query statements using the script engine: The message subscription association processing service routes the message event to the corresponding subscription, and determines the association rule script according to the subscription configuration. Further, the script engine is used to execute the association rule script to extract data values ​​from the original message data as query conditions, and a complete SQL query statement is generated by combining the SQL template and the query conditions;

[0113] Step S504. Generate a query plan using the Calcite engine and perform a database query: Input the generated complete SQL query statement into the query module of the message subscription association processing service. The query module uses the Apache Calcite engine to convert the SQL query statement into a query plan for various databases, and requests database data through the query plan to obtain the associated data of the original message data. The database can be a relational database or a NoSql database, such as an ElasticSearch database, a Mongodb database, a redis database, etc.

[0114] Step S505. Distribute the message data after association processing: After obtaining the associated data of the original message data, the associated data can be supplemented to the original message data. The supplemented message data can be subsequently filtered and distributed through subscription configuration, such as being pushed to a subscription service system, or synchronized to a target database.

[0115] In this example, database support triggers are not required for data association queries, and the associated data obtained can come from different databases, thereby achieving data synchronization of heterogeneous databases. In addition, SQL syntax can support associated data queries of multiple databases (relational databases and NoSql databases), and associated data queries can be implemented through associated query configuration, without using a big data framework to encode associated queries, which greatly simplifies associated queries of message data and improves data query efficiency.

[0116] In the data processing method provided in the exemplary implementation of the present disclosure, a rule script and a query template are configured through a preset service interface; when a message subscription request is received through the preset service interface, the original message data corresponding to the message subscription request is obtained; the rule script corresponding to the original message data is determined, and the target field in the original message data is extracted according to the rule script; a target query statement is generated by combining the query template and the target field; the target query statement is used to query the database to obtain the associated data of the original message data. In the present disclosure, by configuring the rule script and the query template, the associated query of the message data is simplified and the data query efficiency is improved.

[0117] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0118] Furthermore, in this exemplary embodiment, a data processing device is also provided. The device can be applied to a terminal device. Figure 6 As shown, the data processing device 600 may include a related query configuration module 610, a message data acquisition module 620, a query condition extraction module 630, a query statement generation module 640 and a related data query module 660, wherein:

[0119] The associated query configuration module 610 is used to configure the rule script and query template through a preset service interface;

[0120] The message data acquisition module 620 is used to acquire the original message data corresponding to the message subscription request when a message subscription request is received through the preset service interface;

[0121] A query condition extraction module 630, used to determine a rule script corresponding to the original message data, and extract a target field in the original message data according to the rule script;

[0122] A query statement generating module 640, configured to generate a target query statement by combining the query template and the target field;

[0123] The associated data query module 660 is used to perform a database query using the target query statement to obtain the associated data of the original message data.

[0124] In an optional implementation, the data processing device 600 further includes:

[0125] The data source configuration module is used to configure the data source through the preset service interface, and the data source includes at least one of a message queue and a database log.

[0126] In an optional implementation, the message data acquisition module 620 includes:

[0127] A data source determination submodule, configured to determine a target data source according to a message type in a message subscription request when a message subscription request is received through the preset service interface;

[0128] The message data acquisition submodule is used to acquire the original message data corresponding to the message subscription request from the target data source.

[0129] In an optional implementation, the query condition extraction module 630 includes:

[0130] A rule script determination submodule, used to determine the rule script corresponding to the original message data according to the message type of the original message data;

[0131] The query condition extraction submodule is used to execute the rule script using a script engine to extract the target field from the original message data.

[0132] In an optional implementation, the database includes at least one of a relational database and a non-relational database; the associated data query module 660 includes:

[0133] An execution plan generation submodule, used to convert the target query statement into a query plan of the database using a query engine;

[0134] The associated data query submodule is used to query the database based on the query plan to obtain the associated data of the original message data.

[0135] In an optional implementation, the data processing device 600 further includes:

[0136] The associated data cache module is used to cache the associated data of the original message data and configure a mapping relationship between the associated data and the target field.

[0137] In an optional implementation, the data processing device 600 further includes:

[0138] A message data combining module, used for combining the original message data and the associated data of the original message data to obtain target message data;

[0139] The message data push module is used to push the target message data to the corresponding message subscription system.

[0140] In an optional implementation, the query template in the data processing device 600 is a SQL template.

[0141] The specific details of each module in the above data processing device have been described in detail in the corresponding data processing method, so they will not be repeated here.

[0142] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of this specification is stored. In some possible embodiments, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to cause the electronic device to perform the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "exemplary method" section of this specification. The program product may adopt a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, an apparatus or a device or used in combination with it.

[0143] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0146] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0147] The exemplary embodiment of the present disclosure also provides an electronic device capable of implementing the above method. Figure 7 hereinafter, an electronic device 700 according to this exemplary embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0148] like Figure 7 As shown, the electronic device 700 may be in the form of a general-purpose computing device. The components of the electronic device 700 may include, but are not limited to: at least one processing unit 710, at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

[0149] The storage unit 720 stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 executes the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can execute Figures 2 to 5 The method steps in .

[0150] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 721 and / or a cache memory unit 722 , and may further include a read-only memory unit (ROM) 723 .

[0151] The storage unit 720 may also include a program / utility 724 having a set (at least one) of program modules 725, such program modules 725 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0152] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0153] The electronic device 700 may also communicate with one or more external devices 800 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 7100. As shown, the network adapter 7100 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0154] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the exemplary implementation of the present disclosure.

[0155] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0156] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0157] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Configure rule scripts and query templates through preset service interfaces; When a message subscription request is received through the preset service interface, obtaining original message data corresponding to the message subscription request; Determine a rule script corresponding to the original message data, and extract a target field in the original message data according to the rule script; Combining the query template and the target field to generate a target query statement; Performing a database query using the target query statement to obtain associated data of the original message data, and combining the original message data and the associated data of the original message data to obtain target message data; Filtering the target message data, and pushing the target message data to a corresponding message subscription system, so that the message subscription system performs data processing under different data dimensions based on the target message data; Different types of relational data are stored in different types of databases.

2. The data processing method according to claim 1, characterized in that: Before obtaining the original message data corresponding to the message subscription request, the method further includes: A data source is configured through the preset service interface, and the data source includes at least one of a message queue and a database log.

3. The data processing method according to claim 2, characterized in that: When a message subscription request is received through the preset service interface, obtaining original message data corresponding to the message subscription request includes: When a message subscription request is received through the preset service interface, determining a target data source according to a message type in the message subscription request; The original message data corresponding to the message subscription request is obtained from the target data source.

4. The data processing method according to claim 1, characterized in that: The determining the rule script corresponding to the original message data, and extracting the target field in the original message data according to the rule script, includes: Determine a rule script corresponding to the original message data according to the message type of the original message data; The rule script is executed by a script engine to extract the target field from the original message data.

5. The data processing method according to claim 1, characterized in that: The database includes at least one of a relational database and a non-relational database; the database query is performed using the target query statement to obtain the associated data of the original message data, including: Using a query engine to convert the target query statement into a query plan for the database; The database is queried based on the query plan to obtain associated data of the original message data.

6. The data processing method according to claim 1, characterized in that: After obtaining the associated data of the original message data, the method further includes: The associated data of the original message data is cached, and a mapping relationship between the associated data and the target field is configured.

7. The data processing method according to any one of claims 1 to 6, characterized in that: The query template is a SQL template.

8. A data processing device, characterized in that: include: The associated query configuration module is used to configure rule scripts and query templates through the preset service interface; A message data acquisition module, configured to acquire original message data corresponding to a message subscription request when a message subscription request is received through the preset service interface; A query condition extraction module, used to determine the rule script corresponding to the original message data, and extract the target field in the original message data according to the rule script; A query statement generating module, used to generate a target query statement by combining the query template and the target field; A related data query module, used to perform a database query using the target query statement to obtain the related data of the original message data, and combine the original message data and the related data of the original message data to obtain the target message data; Filtering the target message data, and pushing the target message data to a corresponding message subscription system, so that the message subscription system performs data processing under different data dimensions based on the target message data; Different types of relational data are stored in different types of databases.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images