Data bridging method and device, electronic equipment and storage medium

Abstract

The invention relates to a data bridging method and device, electronic equipment and a storage medium, and is applied to the technical field of data processing.The method comprises the steps that according to the type of a message sent by a client, a structured query language rule corresponding to the message is obtained; matching the attribute information of the message with the attribute information in the structured query language rule; if the matching is successful, judging whether the structured query language rule comprises a WHERE clause or not; if not, screening the message based on a SELECT field in the structured query language rule to obtain target information; if yes, screening the message based on the WHERE clauses to obtain a WHERE clause screening message; based on a SELECT field in the structured query language rule, screening the WHERE clause screening message to obtain target information; and bridging the target information to a data source corresponding to the structured query language rule. The flexibility and maintainability of the system can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data bridging method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of big data and Internet of Things technologies, real-time data processing has become increasingly important. In many application scenarios, data distributed in different devices or systems need to be aggregated, processed, and analyzed in real time in order to make quick responses and decisions. For example, in the fields of smart manufacturing, smart transportation, and smart cities, the real-time and accuracy of data are crucial to the operation and management of the system.

[0003] The data gateway is a key component for data aggregation and processing. It is responsible for collecting data from various data sources, performing preliminary processing, and transmitting the data to the backend system. Traditional systems directly use developer coding to send the data received by the data gateway to the message middleware to achieve real-time data processing and transmission. Business platforms are usually operated by independent teams, and decoupling and efficiently connecting the two is the key to rapid delivery and iterative upgrades.

[0004] Traditional data bridging methods require developers to write code to handle the data transmission and conversion process from the data source to the target system. This approach usually requires detailed coding and debugging, and there is a high degree of coupling between the data gateway and the message middleware. Every business change may require modification and redeployment of the code, increasing the complexity and maintenance cost of the system. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a data bridging method, device, electronic device and storage medium.

[0006] According to a first aspect of the present application, a data bridging method is provided, comprising:

[0007] Receive a message sent by a client, and obtain a structured query language rule corresponding to the message according to the type of the message;

[0008] Matching the attribute information of the message with the attribute information in the structured query language rule;

[0009] If the match is successful, determining whether the structured query language rule includes a WHERE clause;

[0010] If the structured query language rule does not include a WHERE clause, filtering the message based on the SELECT field in the structured query language rule to obtain target information;

[0011] If the structured query language rule includes a WHERE clause, filtering the message based on the WHERE clause to obtain a WHERE clause filtered message;

[0012] Based on the SELECT field in the structured query language rule, the WHERE clause filtering message is filtered to obtain target information;

[0013] The target information is bridged to the data source corresponding to the structured query language rule in real time.

[0014] Optionally, before receiving the message sent by the client, the method further includes:

[0015] Get the data source name, data source type and data source property information set by the user;

[0016] Verify whether a connection instance can be established with a target data source according to the data source type and the data source attribute information;

[0017] If a connection instance can be established with the target data source, a connection instance is established with the target data source according to the data source type and the data source attribute information, and a data source name and a connection instance of the target data source are stored;

[0018] Returns information about the established connection instance to the client.

[0019] Optionally, before receiving the message sent by the client, the method further includes:

[0020] Obtaining a rule name set by a user, a monitored message type, a structured query language rule, an output data source, and attribute information of the structured query language rule; wherein the attribute information of the structured query language rule is determined according to the output data source;

[0021] The rule name, the message type, the structured query language rule, the output data source and the attribute information of the structured query language rule are parsed and stored.

[0022] Optionally, matching the attribute information of the message with the attribute information in the structured query language rule includes:

[0023] The subject of the message is matched with the subject in the structured query language rule.

[0024] According to a second aspect of the present application, a data bridging device is provided, comprising:

[0025] A message receiving module, used to receive messages sent by the client;

[0026] A structured query language rule determination module, used to obtain a structured query language rule corresponding to the message according to the type of the message;

[0027] A matching module, used for matching the attribute information of the message with the attribute information in the structured query language rule;

[0028] A WHERE clause judgment module, used to judge whether the structured query language rule includes a WHERE clause if the match is successful;

[0029] A first screening module, configured to screen the message based on a SELECT field in the structured query language rule to obtain target information if the structured query language rule does not include a WHERE clause;

[0030] A second screening module is used for screening the message based on the WHERE clause if the structured query language rule includes a WHERE clause, to obtain a WHERE clause screened message;

[0031] A third screening module, configured to screen the WHERE clause screening message based on the SELECT field in the structured query language rule to obtain target information;

[0032] The bridging module is used to bridge the target information to the data source corresponding to the structured query language rule in real time.

[0033] Optionally, the device further comprises:

[0034] The data source information acquisition module is used to obtain the data source name, data source type and data source attribute information set by the user;

[0035] A connection instance verification module, used to verify whether a connection instance can be established with a target data source according to the data source type and the data source attribute information;

[0036] A connection instance establishment module, configured to establish a connection instance with the target data source according to the data source type and the data source attribute information if a connection instance can be established with the target data source;

[0037] A connection instance storage module, used to store the data source name and connection instance of the target data source;

[0038] The established information return module is used to return the information of the established connection instance to the client.

[0039] Optionally, the device further comprises:

[0040] A module for acquiring information related to structured query language rules, used to acquire a rule name set by a user, a monitored message type, a structured query language rule, an output data source, and attribute information of the structured query language rule; wherein the attribute information of the structured query language rule is determined according to the output data source;

[0041] The parsing and storage module is used to parse and store the rule name, the message type, the structured query language rule, the output data source and the attribute information of the structured query language rule.

[0042] Optionally, the matching module is specifically configured to match the subject of the message with a subject in the structured query language rule.

[0043] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, wherein the processor is configured to execute a computer program stored in a memory, wherein the computer program implements the method described in the first aspect when executed by the processor.

[0044] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0045] According to a fifth aspect of the present application, a computer program product is provided. When the computer program product is run on a computer, the computer is enabled to execute the method described in the first aspect.

[0046] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0047] Introducing a built-in real-time screening and processing mechanism in the data gateway, that is, pre-storing structured query language rules, and filtering the messages sent by the client in real time before transmitting them according to the structured query language rules, which improves the efficiency of data processing and reduces additional coding work and system complexity. Without changing the existing logic of the data gateway, by decoupling the data gateway from the message middleware, the maintenance and expansion of the system is simplified, and the flexibility and maintainability of the system are improved. This application can be flexibly configured and managed, which simplifies the system integration and management process and reduces maintenance costs and the complexity of system expansion. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 A flow chart of a data bridging method in an embodiment of the present application;

[0051] Figure 2 A schematic diagram of the structure of a data bridging device in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0053] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0054] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0055] The data bridging method of the embodiment of the present application is applied to a data gateway. Before the data gateway bridges the message published by the client, the user of the data gateway can first configure the data gateway, including: adding a data source and adding a structured query language rule.

[0056] Optionally, the method for adding a data source can be:

[0057] The user enters a name for the new data source, selects the type of data source (such as database, Kafka, etc.), and configures related attribute information based on the selected data source type, such as the connection information of the user-configured database, the broker address of Kafka, etc.)

[0058] After the user settings are completed, the data gateway can obtain the data source name, data source type, and data source attribute information set by the user. According to the data source type and data source attribute information, verify whether a connection instance can be established with the target data source to ensure that the data source connection is correct. For example, use the "ping" command to verify whether a connection instance can be established with the target data source. If a connection instance can be established with the target data source, a connection instance is established with the target data source according to the data source type and data source attribute information, and the data source name and connection instance of the target data source are stored. In addition, the information of the established connection instance is returned to the client, so that the user can know that the connection instance has been established.

[0059] Optionally, the method of adding a structured query language rule may be:

[0060] The user can set the rule name of the added structured query language rule and select the type of message to be monitored, such as publishing messages, subscribing messages, unsubscribing, establishing connections, heartbeats, etc. The user sets the specific content of the added structured query language rule. For example, the added structured query language rule is: select msg from "test" where clientId = 1 and level>5, which means that the data gateway monitors messages whose client id is 1 and the topic is test. The user selects the output data source, which is the data source added in the above method of adding a data source. The user selects the output data source, that is, adds the corresponding structured query language rule for the data source selected by the user. The user can also add custom attributes based on different data sources, such as Kafka, including partition key, topic and other attributes.

[0061] After the user settings are completed, the data gateway can obtain the rule name set by the user, the monitored message type, the structured query language rule, the output data source, and the attribute information of the structured query language rule. Among them, the attribute information of the structured query language rule is determined according to the output data source, that is, the user determines the attribute information of the structured query language rule according to the output data source. Parse and store the rule name, message type, structured query language rule, output data source, and attribute information of the structured query language rule. ) That is, the data gateway parses the above information into a language that the data gateway can recognize, for example, it can be parsed using the LR algorithm, etc.

[0062] See also Figure 1 , Figure 1 A flow chart of the data bridging method in an embodiment of the present application may include the following steps:

[0063] Step S102: receiving a message sent by the client, and acquiring a structured query language rule corresponding to the message according to the type of the message.

[0064] As mentioned above, when adding a structured query language rule, the type of message to be monitored can be set. Therefore, after receiving a message sent by a client, the structured query language rule corresponding to the message can be obtained according to the type of the message, so as to filter the message in real time according to the structured query language rule.

[0065] Step S104: Match the attribute information of the message with the attribute information in the structured query language rule.

[0066] Different data sources may correspond to different attribute information. For example, Kafka includes attributes such as partition key and topic. Therefore, the structured query language rules corresponding to different data sources also correspond to different attribute information. The attribute information of the message is matched with the attribute information in the structured query language rule. If it does not match, it means that the message cannot be bridged to the data source corresponding to the structured query language rule, and the process ends. If the match is successful, continue to step S106 for further judgment. Optionally, the attribute information of the message and the attribute information in the structured query language rule are both topics, and the topic of the message can be matched with the topic in the structured query language rule.

[0067] Step S106, determining whether the structured query language rule includes a WHERE clause.

[0068] If the structured query language rule does not include a WHERE clause, step S108 is executed; if the structured query language rule includes a WHERE clause, step S110 is executed.

[0069] Step S108, based on the SELECT field in the structured query language rule, the message is screened to obtain target information.

[0070] Step S110, filtering the message based on the WHERE clause to obtain the WHERE clause filtered message.

[0071] It should be noted that after filtering the message based on the WHERE clause, if the WHERE clause filtered message is empty, step S112 is no longer executed and the process ends. If the WHERE clause filtered message is not empty, step S112 is continued to be executed for further filtering.

[0072] Step S112, based on the SELECT field in the structured query language rule, the WHERE clause filtering message is filtered to obtain target information.

[0073] Step S114: bridging the target information to the data source corresponding to the structured query language rule in real time.

[0074] The data bridging method of the embodiment of the present application introduces a built-in real-time screening and processing mechanism in the data gateway, that is, the structured query language rules are pre-stored, and the messages sent by the client are screened in real time according to the structured query language rules before transmission, which improves the efficiency of data processing and reduces additional coding work and system complexity. Without changing the existing logic of the data gateway, by decoupling the data gateway and the message middleware, the maintenance and expansion of the system are simplified, and the flexibility and maintainability of the system are improved. The present application can be flexibly configured and managed, which simplifies the integration and management process of the system and reduces the maintenance cost and complexity of system expansion.

[0075] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0076] Corresponding to the above method embodiment, the present application embodiment also provides a data bridging device, see Figure 2 , the data bridging device 200 comprises:

[0077] The message receiving module 202 is used to receive messages sent by the client;

[0078] The structured query language rule determination module 204 is used to obtain the structured query language rule corresponding to the message according to the type of the message;

[0079] A matching module 206, used to match the attribute information of the message with the attribute information in the structured query language rule;

[0080] A WHERE clause determination module 208, for determining whether the structured query language rule includes a WHERE clause if the match is successful;

[0081] A first screening module 210, configured to screen the message based on the SELECT field in the structured query language rule to obtain target information if the structured query language rule does not include a WHERE clause;

[0082] A second screening module 212 is configured to screen messages based on the WHERE clause if the structured query language rule includes a WHERE clause, and obtain messages screened by the WHERE clause;

[0083] The third screening module 214 is used to screen the WHERE clause screening message based on the SELECT field in the structured query language rule to obtain target information;

[0084] The bridging module 216 is used to bridge the target information to the data source corresponding to the structured query language rule in real time.

[0085] Optionally, the data bridging device 200 further includes:

[0086] The data source information acquisition module is used to obtain the data source name, data source type and data source attribute information set by the user;

[0087] A connection instance verification module is used to verify whether a connection instance can be established with a target data source based on the data source type and data source attribute information;

[0088] A connection instance establishment module, used to establish a connection instance with the target data source according to the data source type and data source attribute information if a connection instance can be established with the target data source;

[0089] A connection instance storage module, used to store the data source name and connection instance of the target data source;

[0090] The established information return module is used to return the information of the established connection instance to the client.

[0091] Optionally, the data bridging device 200 further includes:

[0092] The module for acquiring information related to structured query language rules is used to acquire the rule name set by the user, the type of monitored message, the structured query language rule, the output data source and the attribute information of the structured query language rule; wherein the attribute information of the structured query language rule is determined according to the output data source;

[0093] The parsing and storage module is used to parse and store the rule name, message type, structured query language rule, output data source and attribute information of the structured query language rule.

[0094] Optionally, the matching module 206 is specifically configured to match the subject of the message with the subject in the structured query language rule.

[0095] The specific details of each module or unit in the above device have been described in detail in the corresponding method, so they will not be repeated here.

[0096] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0097] The present application also provides an electronic device, see Figure 3 , Figure 3 The electronic device of the embodiment of the present application includes: a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304;

[0098] Memory 303, used for storing computer programs;

[0099] The processor 301 is used to implement the above data bridging method when executing the program stored in the memory 303.

[0100] It should be noted that the communication bus 304 mentioned in the above electronic device can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 304 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0101] The communication interface 302 is used for communication between the electronic device and other devices.

[0102] The memory 303 may include a RAM (Random Access Memory), or may include a non-volatile memory, such as at least one disk memory. Optionally, the memory 303 may also be at least one storage device located away from the aforementioned processor.

[0103] The above-mentioned processor 301 can be a general-purpose processor, including: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0104] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data bridging method is implemented.

[0105] It should be noted that the computer-readable storage medium shown in the present application may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency, etc., or any suitable combination of the above.

[0106] In an embodiment of the present application, a computer program product is also provided. When the computer program product is run on a computer, the computer executes the above-mentioned data bridging method.

[0107] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0108] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data bridging method, characterized in that: The method comprises: Receive a message sent by a client, and obtain a structured query language rule corresponding to the message according to the type of the message; Matching the attribute information of the message with the attribute information in the structured query language rule; If the match is successful, determining whether the structured query language rule includes a WHERE clause; If the structured query language rule does not include a WHERE clause, filtering the message based on the SELECT field in the structured query language rule to obtain target information; If the structured query language rule includes a WHERE clause, filtering the message based on the WHERE clause to obtain a WHERE clause filtered message; Based on the SELECT field in the structured query language rule, the WHERE clause filtering message is filtered to obtain target information; The target information is bridged to the data source corresponding to the structured query language rule in real time.

2. The method according to claim 1, characterized in that: Before receiving the message sent by the client, the method further includes: Get the data source name, data source type and data source property information set by the user; Verify whether a connection instance can be established with a target data source according to the data source type and the data source attribute information; If a connection instance can be established with the target data source, a connection instance is established with the target data source according to the data source type and the data source attribute information, and a data source name and a connection instance of the target data source are stored; Returns information about the established connection instance to the client.

3. The method according to claim 1, characterized in that Before receiving the message sent by the client, the method further includes: Obtaining a rule name set by a user, a monitored message type, a structured query language rule, an output data source, and attribute information of the structured query language rule; wherein the attribute information of the structured query language rule is determined according to the output data source; The rule name, the message type, the structured query language rule, the output data source and the attribute information of the structured query language rule are parsed and stored.

4. The method according to claim 1, characterized in that: The matching of the attribute information of the message with the attribute information in the structured query language rule includes: The subject of the message is matched with the subject in the structured query language rule.

5. A data bridging device, characterized in that: The device comprises: A message receiving module, used to receive messages sent by the client; A structured query language rule determination module, used to obtain a structured query language rule corresponding to the message according to the type of the message; A matching module, used for matching the attribute information of the message with the attribute information in the structured query language rule; A WHERE clause judgment module, used to judge whether the structured query language rule includes a WHERE clause if the match is successful; A first screening module, configured to screen the message based on a SELECT field in the structured query language rule to obtain target information if the structured query language rule does not include a WHERE clause; A second screening module is used for screening the message based on the WHERE clause if the structured query language rule includes a WHERE clause, to obtain a WHERE clause screened message; A third screening module, configured to screen the WHERE clause screening message based on the SELECT field in the structured query language rule to obtain target information; The bridging module is used to bridge the target information to the data source corresponding to the structured query language rule in real time.

6. The device according to claim 5, characterized in that The device also includes: The data source information acquisition module is used to obtain the data source name, data source type and data source attribute information set by the user; A connection instance verification module, used to verify whether a connection instance can be established with a target data source according to the data source type and the data source attribute information; A connection instance establishment module, configured to establish a connection instance with the target data source according to the data source type and the data source attribute information if a connection instance can be established with the target data source; A connection instance storage module, used to store the data source name and connection instance of the target data source; The established information return module is used to return the information of the established connection instance to the client.

7. The device according to claim 5, characterized in that The device also includes: The module for acquiring information related to structured query language rules is used to acquire the rule name set by the user, the type of monitored message, the structured query language rule, the output data source and the attribute information of the structured query language rule; wherein the attribute information of the structured query language rule is determined according to the output data source; The parsing and storage module is used to parse and store the rule name, the message type, the structured query language rule, the output data source and the attribute information of the structured query language rule.

8. The device according to claim 5, characterized in that The matching module is specifically configured to match the subject of the message with the subject in the structured query language rule.

9. An electronic device, characterized in that: include: A processor, wherein the processor is used to execute a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images