Data pushing method and device, computer device and storage medium

Abstract

The application relates to the technical field of data processing, and discloses a data pushing method and device, computer equipment and a storage medium, the method of which comprises the following steps: receiving pushing demand information; obtaining an examination and approval result of the pushing demand information; if the examination and approval result is examination and approval passing, generating a pushing configuration page according to the pushing demand information; determining a target data source, a target pushing interface and a data pushing rule through the pushing configuration page; obtaining pushing data from the target data source according to the pushing demand information, and pushing the pushing data to the target pushing interface through the data pushing rule, so that a data receiver receives the pushing data through the target pushing interface. The data pushing method simplifies a data pushing process, improves the applicability to various data pushing demands, reduces the management and maintenance difficulty of a data pushing program, and improves the data pushing efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a data pushing method and device, a computer device and a storage medium. BACKGROUND

[0002] In the era of big data, enterprises need to push data in many scenarios. On the one hand, enterprises need to push data to relevant departments to facilitate the relevant departments to understand the business development of local enterprises from the business data of local enterprises, and the relevant supervisory departments can also investigate and deal with illegal events of enterprises through business data. In order to meet the needs of relevant departments, each enterprise needs to set up a special person for connection, which consumes a lot of time and effort. On the other hand, data pushing between enterprises and cooperative enterprises is also inevitable. Since the data pushing needs of cooperative enterprises are different, a customized data pushing program or a specially modified data pushing program needs to be developed for each cooperative enterprise. With more and more cooperative enterprises, the management and maintenance of data pushing programs become more and more complicated.

[0003] When existing enterprises push data, on the one hand, business data is the core data of enterprises. In order to avoid information leakage, strict pushing procedures need to be followed when pushing data to relevant departments or cooperative enterprises, which inevitably leads to complex and lengthy procedures and consumes a lot of connection time. On the other hand, the relevant departments or cooperative enterprises are a separate demand side, and the data pushing needs are basically fixed. For enterprises without software development capabilities, there will be a lot of repetitive connection work, which reduces the data pushing efficiency. At the same time, for enterprises with software development capabilities, since there is a technical isolation between the business data of enterprises, data sharing cannot be carried out. When connecting each cooperative enterprise, the data pushing program has the characteristics of customization, and the expansibility of the data pushing program is poor. When there are differences in the data source, data demand or pushing interface standard of cooperative enterprises, the pushing needs cannot be met, which leads to an increase in the management and maintenance difficulty of the data pushing program and a reduction in the data pushing efficiency.

[0004] Therefore, there is an urgent need for a data pushing method that can meet different pushing needs, reduce management difficulty and improve pushing efficiency. SUMMARY

[0005] Therefore, it is necessary to provide a data pushing method, device, computer device and storage medium to solve the problems of complex pushing process, poor system applicability and low pushing efficiency.

[0006] A data pushing method comprises:

[0007] receiving pushing demand information;

[0008] obtain an approval result of the push demand information;

[0009] generate a push configuration page according to the push demand information if the approval result is approval;

[0010] determine a target data source, a target push interface and a data push rule through the push configuration page;

[0011] obtain push data from the target data source according to the push demand information, and push the push data to the target push interface through the data push rule, so that a data receiver receives the push data through the target push interface.

[0012] A data push device comprises:

[0013] a push demand receiving module configured to receive push demand information;

[0014] an approval result obtaining module configured to obtain an approval result of the push demand information;

[0015] a configuration page generating module configured to generate a push configuration page according to the push demand information if the approval result is approval;

[0016] a push configuration determining module configured to determine a target data source, a target push interface and a data push rule through the push configuration page;

[0017] a data push module configured to obtain push data from the target data source according to the push demand information, and push the push data to the target push interface through the data push rule, so that a data receiver receives the push data through the target push interface.

[0018] A computer device comprises a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, and the processor implements the above data push method when executing the computer readable instructions.

[0019] One or more readable storage media storing computer readable instructions, and the computer readable instructions are executed by one or more processors to make the one or more processors execute the above data push method.

[0020] The data pushing method, device, computer device and storage medium, the method of which comprises the following steps: receiving pushing demand information; obtaining an approval result of the pushing demand information; if the approval result is approval, generating a pushing configuration page according to the pushing demand information; determining a target data source, a target pushing interface and a data pushing rule through the pushing configuration page; obtaining pushing data from the target data source according to the pushing demand information, and pushing the pushing data to the target pushing interface through the data pushing rule, so that a data receiver receives the pushing data through the target pushing interface. The data pushing method of the present application realizes data pushing based on online configuration of data sources, pushing interfaces and data pushing rules on a data pushing platform, simplifies the data pushing process, improves the applicability to various data pushing demands, reduces the management and maintenance difficulty of the data pushing program, and improves the data pushing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1 is a flowchart of a data pushing method in an embodiment of the present application;

[0023] Figure 2 is a structural schematic diagram of a data pushing device in an embodiment of the present application;

[0024] Figure 3 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The technical solutions of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0026] In an embodiment, as shown in Figure 1 , a data pushing method is provided, comprising the following steps S10-S50.

[0027] S10, receiving pushing demand information.

[0028] Understandably, the data sender of the data push is the current user enterprise, and the data receiver includes relevant departments and cooperative enterprises which have data push demand with the current user enterprise. The data push method of the embodiment is realized based on a data push platform. A server of the data push platform provides data push service for the data sender and the data receiver, so as to push the data of the data sender to the data receiver. The push demand information is the description information about the pushed data provided by the data receiver.

[0029] The push demand information includes demand name, demand applicant, demand application time, counterpart, demand priority, demand background, demand field, data risk level, expected completion time, and proof material attachment, etc. For example, when the data receiver is a relevant department, the push demand information provided by the relevant department includes standard demand background, address of each push interface, request parameter, return parameter, and official letter. After receiving the push demand information, the server sends the push demand information of the data receiver to the data sender and initiates approval, so as to provide basis for online demand approval of the data sender. The push demand information can be repeatedly edited before initiating approval. The server pre-establishes a push demand list, and saves the push demand information into the push demand list, so as to facilitate searching and reuse of the push demand information.

[0030] S20, obtaining an approval result of the push demand information.

[0031] Understandably, the server sends the push demand information of the data receiver to the data sender and initiates approval, and the data sender approves the push demand information and returns an approval result. The data sender creates an approval workflow, and a callback event of an approval node in the approval workflow triggers a message reminder to remind the corresponding approval personnel to timely enter an approval page for approval. The server obtains the approval result of the push demand information. If the approval is passed, the data push demand takes effect. If the approval is not passed, the data push demand is rejected, and needs to be re-initiated.

[0032] In the embodiment, the data push platform can not only record information such as demand party, demand background, demand content, and expected completion time to realize online management of data push demand, but also can perform online approval of the data push demand. The data push platform collects data push demand of the data receiver (such as a relevant department) for separate analysis, the data sender (such as a local enterprise) makes internal approval according to the official letter content of the relevant department, and then performs secondary confirmation with the relevant department, so as to directly supplement and confirm the field online without redundant communication.

[0033] S30, if the approval result is approval passed, generating a push configuration page according to the push demand information.

[0034] Understandably, the push configuration page is a page for configuring each node in the data push task flow. If the approval result is approval, the push configuration page is generated to enable the data sender to input push task configuration information in the push configuration page according to push requirement information.

[0035] In this embodiment, the flow of the data push task includes obtaining initial push data from a data source of the data sender, filtering, parameter mapping and encryption processing of the initial push data to obtain to-be-pushed data, and outputting the to-be-pushed data to the data receiver through a push interface. The push configuration page includes a data source configuration page, a data filtering rule configuration page, a parameter mapping rule configuration page, an encryption rule configuration page and a push interface configuration page.

[0036] S40, determining a target data source, a target push interface and a data push rule through the push configuration page.

[0037] Understandably, the data source is the source of the pushed data, such as a database, a message queue, a data interface and a data file, and contains configuration information required for obtaining data from the data source. The source data is data obtained from the data source, such as data queried from a database. The push interface is an interface through which the data receiver receives data, such as an http interface, a Dubbo interface and a message queue, and contains configuration information required for outputting data through the push interface.

[0038] The data sender inputs push task configuration information in the push configuration page according to push requirement information, and the server creates a data push task according to the push task configuration information. The server pre-establishes a push task list, saves the push task configuration information into the push task list, and facilitates searching and reusing the push task configuration information.

[0039] In this embodiment, one data push task corresponds to one data source, one push interface and one set of data push rules. The target data source is the data source configured in the current data push task, the target push interface is the push interface configured in the current data push task, and the data push rule is a rule for processing initial push data obtained from the target data source, such as a data filtering rule, a parameter mapping rule and an encryption rule.

[0040] The data push task includes, in addition to the basic information of the task (such as the push name, encryption key and push frequency), the push task configuration information of the data source, push interface, filter (filtering rule), data converter (parameter mapping rule) and encrypter (encryption rule). The server of the data push platform abstracts each push configuration page into the components of the data source, push interface, filter, data converter and encrypter, realizes customization and free combination of the components in a dynamic online configuration manner, and thus creates various data push tasks to meet various data push requirements. Meanwhile, the server designs an extension interface for the data source, push interface, filter, data converter and encrypter, and enriches the functions of the data push platform through the extension interface.

[0041] S50, acquiring push data from the target data source according to the push requirement information, and pushing the push data to the target push interface through the data push rule, so that the data receiver receives the push data through the target push interface.

[0042] Understandably, the data push task is in an unstarted state after being created, and the data push process is triggered when new data is generated in the target data source after the push is started. If the current user enterprise needs to modify the push task configuration information of the data push task, the data push task can be paused, and the data push task is started again after the modification is completed. The start and stop and modification of the data push task take effect immediately, without the need to restart the data push platform.

[0043] After the first completion of the data push connection between the data sender and the data receiver, the data push task can be managed online. When the data receiver proposes to stop the data push requirement, the data push can be stopped directly through the pause button in the management interface of the data push platform. When the data receiver proposes to resume the data push requirement, the push can be performed again through the start button in the management interface of the data push platform, to realize real-time starting or pausing of the data push task. The server of the data push platform performs data statistics according to the time, requirement, data source, push interface and other dimensions, and displays the statistical results in a chart, to realize real-time monitoring of the data indexes such as the push success rate and average time consumption, and to initiate a warning notification through an email or short message when the data indexes are abnormal.

[0044] In this embodiment, after the data push process is triggered, the server obtains initial push data matched with the push requirement information from the data source of the data sender, obtains push data by filtering, parameter mapping and encryption processing on the initial push data according to the data push rule, and outputs the push data to the data receiver through the push interface, so that the data receiver receives the push data through the target push interface.

[0045] The embodiment receives push demand information, obtains an approval result of the push demand information, generates a push configuration page according to the push demand information if the approval result is approval, determines a target data source, a target push interface and a data push rule through the push configuration page, obtains push data from the target data source according to the push demand information, and pushes the push data to the target push interface through the data push rule, so that a data receiver receives the push data through the target push interface. The data push method of the embodiment realizes online configuration of a data source, a push interface and a data push rule for data push based on a data push platform, simplifies a data push process, improves the applicability to various data push demands, reduces the management and maintenance difficulty of a data push program, and improves the data push efficiency.

[0046] When the data sender is a local enterprise and the data receiver is a related department, the data push method of the embodiment is beneficial to both parties. On the one hand, for the related department, the data push efficiency can be improved, and the related department only needs to provide push demand information (such as a public letter and the address of the push interface) to provide the data push platform to interface the business data of the local enterprise. The interfacing time can also be reduced. It took more than a few months to complete the interfacing from the beginning of the interfacing to the completion of the interfacing by the enterprise in the past. Now the interfacing can be completed within one day. On the other hand, for the local enterprise, first, the technical threshold is reduced, and the local enterprise does not need to have software development capability, and can realize data push through online configuration of the push configuration page of the data push platform. Second, the time and labor cost is reduced. The push program code does not need to be repeatedly developed, and data push can be realized through online configuration of the push configuration page of the data push platform directly. There is no need to set up a person to interface the data push demand of the related department. The data push platform only serves as a data pipeline and does not store data, avoiding the problem of leakage of business data of the user enterprise. Third, the data push platform realizes centralized management of various data push demands, online approval of the demands, seamless change of the data source, real-time start and stop of the push task, online monitoring and early warning, greatly reducing the management and maintenance difficulty and improving the demand interfacing efficiency, thereby reducing the data push cost. For various unexpected situations of the related department, such as suspension of the data push task, the push can be immediately suspended, or after suspension, the data push task can be immediately started. Finally, the communication efficiency is improved. The local enterprise can perform standardized interfacing and online communication with the related department according to the actual situation of the respective enterprise through the data push platform, without offline communication. The online communication content is confirmed by both parties and is archived in real time.

[0047] Optionally, the push configuration page includes a data source configuration page; and in step S40, the target data source, the target push interface and the data push rule are determined through the push configuration page, including:

[0048] S401, receiving data source parameter configuration information of a plurality of data sources through the data source configuration page, and generating a data source list according to the data source parameter configuration information;

[0049] S402, receiving a data source selection instruction through the data source configuration page, and selecting the target data source from the data source list according to the data source selection instruction.

[0050] Understandably, the data push platform abstracts data sources into data source components based on the data source configuration page, and supports obtaining data from multiple types of data sources. The server receives data source parameter configuration information of a plurality of data sources through the data source configuration page, and pre-establishes a data source list according to the data source parameter configuration information, so as to select a target data source from the data source list according to the data push demand to create a data push task.

[0051] The data source includes several types, for example: Kafka, database, data interface and data file, and different data sources are distinguished by the type field of the data source. Different data sources correspond to different data source parameter configuration information, and the data push platform reserves a data source expansion interface to support expansion of the data source type. When the data source is of the Kafka type, the data source parameter configuration information includes the Kafka cluster address, the message topic, the message grouping, the number of consumer instances and the message body structure. When the data source is of the database type, the data source parameter configuration information includes the database address, the database name, the account, the password, the data query sql statement, the data query frequency and the data structure. When the data source is of the data interface type, the data source parameter configuration information includes the interface address, the signature key, the parameter format, the request header parameter and the request body data structure. When the data source is of the data file type, the data source parameter configuration information includes the file directory, the file format and the data structure. Each type of data source needs to configure the data structure, and the field name and field description information need to be configured in the data structure.

[0052] When creating a data push task, a data source selection instruction is received through the data source configuration page, a data source is selected from the data source list according to the data source selection instruction to subscribe to the data source as a target data source, and the data push process is triggered when new data of the target data source is generated. After the data push task is started, when the data source changes, the data source parameter configuration information and the data source selection instruction can be updated through the data source configuration page to modify the data source, and the modified data source takes effect immediately without the need to restart the data push platform.

[0053] The embodiment abstracts the data source into a data source component based on a data source configuration page, generates a data source list through data source parameter configuration information of multiple data sources, supports extension of data source types for the data source component, and improves the applicability of the data pushing platform. Meanwhile, customization of the data source component is realized in a dynamic online configuration manner, thereby meeting various data pushing requirements, facilitating switching and management of the data source, and improving data pushing efficiency.

[0054] Optionally, the pushing configuration page includes a pushing interface configuration page, and in step S40, the determining the target data source, the target pushing interface, and the data pushing rule through the pushing configuration page includes:

[0055] S403, receiving pushing interface parameter configuration information of multiple pushing interfaces through the pushing interface configuration page, and generating a pushing interface list according to the pushing interface parameter configuration information;

[0056] S404, receiving a pushing interface selection instruction through the pushing interface configuration page, and selecting the target pushing interface from the pushing interface list according to the pushing interface selection instruction.

[0057] Understandably, the data pushing platform abstracts the pushing interface into a pushing interface component based on the pushing interface configuration page, supports pushing data to multiple types of pushing interfaces. The server receives pushing interface parameter configuration information of multiple pushing interfaces through the data source configuration page, and pre-establishes a pushing interface list according to the pushing interface parameter configuration information, so as to select a target pushing interface from the pushing interface list to create a data pushing task according to data pushing requirements.

[0058] The push interface includes several types, for example: Kafka, an http interface, and a database. Different push interfaces are distinguished by a type field of the push interface. Different push interfaces correspond to different push interface parameter configuration information. The data push platform reserves a push interface extension interface to support the extension of the push interface type. When the push interface is of the Kafka type, the push interface parameter configuration information includes a Kafka cluster address, a message topic, and a data structure. When the push interface is of the http interface type, the push interface parameter configuration information includes an interface address, a message header, a parameter list, and a data structure. When the push interface is of the database type, the push interface parameter configuration information includes a database address, a database name, an account, a password, a data insertion SQL statement, and a data structure. Each type of push interface needs to configure a data structure, and the data structure needs to configure parameter name, parameter description, parameter type, whether to be filled in, default value, whether to be encrypted, and the like. In creating a data push task, a push interface configuration page receives a push interface selection instruction, and selects a target push interface from a push interface list according to the push interface selection instruction. After the data push task is started, the server outputs the processed push data to a data receiver through the target push interface according to the data push rule.

[0059] The embodiment abstracts the push interface into a push interface component based on a push interface configuration page, generates a push interface list through push interface parameter configuration information of multiple push interfaces, and supports the extension of the push interface type, thereby improving the applicability of the data push platform. Meanwhile, the push interface component is customized in a dynamic online configuration manner, thereby meeting various data push requirements and improving data push efficiency.

[0060] Optionally, the data push rule includes a data filtering rule, and the push configuration page includes a data filtering rule configuration page. In step S40, the determination of the target data source, the target push interface, and the data push rule through the push configuration page includes:

[0061] S405, receiving data judgment condition information and judgment value information through the data filtering rule configuration page;

[0062] S406, generating the data filtering rule according to the data judgment condition information and the judgment value information.

[0063] Understandably, the data obtained from the target data source needs to be filtered to delete data that does not meet the requirements. The data filtering rule is used to check whether the data obtained from the target data source meets the push requirements. One data push task can configure multiple data filtering rules. Data push will be performed only when multiple data filtering rules are met. The data push platform abstracts the data filtering rule into a filter component based on a data filtering rule configuration page. In an embodiment, the server receives data filtering rule configuration information through the data filtering rule configuration page. The data filtering rule configuration information includes source data information, judgment condition information, and judgment value information. The source data information can be the value of a field in the source data or the value of the source data processed by a rule expression. The judgment condition information refers to the comparison result information of the source data and the judgment value. The judgment condition information can be equal to, not equal to, less than, less than or equal to, greater than, greater than or equal to, contain, empty, or not empty. The judgment value can be a fixed value set in advance, the value of a field in the source data, or the value of the source data processed by a rule expression.

[0064] The rule expression is a piece of text with specific rules, including three rules of literal, reference source data field and custom function, the three rules can be spliced or nested, and the source data can be processed after being parsed by the rule engine. The rule expression can be flexibly used for data filtering and parameter mapping data processing. In the literal rule, the text content is directly filled in, for example, the text "province" is still "province" after being parsed by the rule engine. In the reference source data field rule, the source data field can be referenced by using ${field name}, for example, when address splicing is needed, the rule expression can be filled in as "${province} province ${city} city", province is the field name of the province in the source data, and city is the field name of the city in the source data. If the value of the province field in the source data is "Guangdong" and the value of the city field is "Shenzhen", the data parsed by the rule engine is "Guangdong Province Shenzhen City". The custom function is a function for processing specific data, which is called by the custom rule, for example, functions for obtaining the current time, time formatting, generating serial numbers, etc. The custom function rule uses "#{function name: parameter1, parameter2,...}" to call the custom function. For example, if the original content of the service time field in the source data is "20221022123000" and needs to be converted to "2022-10-22 12:30:00" format, the rule expression can be filled in as "#{TimeFormat: ${serverTime}, yyyy-mm-dd HH24:mi:ss}". Among them, "TimeFormat" is the function name of time formatting, "serverTime" is the field name of the service time in the source data, and "yyyy-mm-dd HH24:mi:ss" is the time format. The data pushing platform has built-in time formatting, obtaining incremental sequence and obtaining random number and many other basic functions, and also provides an extension interface to support the extension of the function library, and even the complex processing of calling the authentication interface to obtain authentication information can be realized by using the custom function.

[0065] The embodiment abstracts the data filtering rule into a filter component based on the data filtering rule configuration page. The filter component can configure multiple data filtering rules for a data pushing task. The data pushing is performed only when the multiple data filtering rules are all satisfied, thereby improving the accuracy of the source data filtering and verification. Meanwhile, the customization of the filter component is realized in a dynamic online configuration manner, thereby meeting various data pushing requirements and improving the data pushing efficiency.

[0066] Optionally, the data pushing rule includes a parameter mapping rule, and the pushing configuration page includes a parameter mapping rule configuration page; in the step S40, the target data source, the target pushing interface and the data pushing rule are determined through the pushing configuration page, including:

[0067] S407, acquire data source field information of the target data source;

[0068] S408, acquire push interface parameter information of the target push interface;

[0069] S409, receive mapping relationship information of the data source field information and the push interface parameter information through the parameter mapping rule configuration page, and generate the parameter mapping rule according to the mapping relationship information.

[0070] Understandably, there is a difference between the data structure of the data source and the data structure of the push interface, so parameter mapping is needed before data pushing. The data pushing platform abstracts the parameter mapping rule into a data converter component based on a parameter mapping rule configuration page. The parameter mapping rule is used to read the data structure of the data source and the data structure of the push interface, and configure the mapping relationship between the data source field and the push interface parameter. A corresponding source data field can be directly selected for the push interface parameter, for example, the push interface parameter "order amount" is directly mapped to the "orderAmount" field of the source data. The push interface parameter can also be processed using a rule expression before mapping, for example, the push interface parameter "order region" can be processed using the rule expression "${province} province ${city} city" before mapping, and mapped to the value "Guangdong" of the province field and the value "Shenzhen" of the city field in the source data. When creating a data pushing task, the parameter mapping rule is generated according to the mapping relationship between the data source field information and the push interface parameter information.

[0071] The embodiment abstracts the parameter mapping rule into a data converter component based on a parameter mapping rule configuration page, realizes customization of the parameter mapping rule component in a dynamic online configuration manner, thereby meeting various data pushing requirements and improving data pushing efficiency. At the same time, the mapping relationship between the data source field and the push interface parameter improves the accuracy of the source data parameter mapping.

[0072] Optionally, the data pushing rule includes an encryption rule, and the push configuration page includes an encryption rule configuration page; in step S40, that is, the target data source, the target push interface and the data pushing rule are determined through the push configuration page, including:

[0073] S410, receive an encrypter selection instruction through the encryption rule configuration page, and select a target encrypter from an encrypter list according to the encrypter selection instruction;

[0074] S411, receive encryption rule configuration information of the target encrypter through the encryption rule configuration page, and configure the encryption rule according to the encryption rule configuration information.

[0075] Understandably, the data receiver can encrypt the data according to the data push requirement and then push the data. The data push platform abstracts the encryption rule into an encryptor component based on an encryption rule configuration page, supports encryption processing using multiple types of encryption modes. The server receives encryption rule configuration information of multiple encryption modes through the encryption rule configuration page, and pre-establishes an encryptor list according to the encryption rule configuration information, so as to select a target encryptor from the encryptor list to create a data push task according to the data push requirement. The encryption rule configuration information of each encryption mode in the encryptor includes an encryption algorithm, an encryption rule, and a ciphertext format. The data push platform internally has encryption modes of commonly used encryption algorithms such as data encryption standard (DES) and public key cryptography system (RSA), and reserves an encryptor extension interface to support extension of the encryption mode of the encryptor. In an embodiment, the server receives an encryptor selection instruction through the encryption rule configuration page, selects a target encryptor from the encryptor list according to the encryptor selection instruction, and configures the encryption rule through the encryption algorithm, the encryption rule, and the ciphertext format of the target encryptor.

[0076] This embodiment abstracts the encryption rule into an encryptor component based on the encryption rule configuration page, generates an encryptor list through encryption rule configuration information of multiple encryption modes, and the encryptor component supports extension of the encryption mode, thereby improving the applicability of the data push platform. At the same time, the customization of the encryptor component is realized in a dynamic online configuration manner, thereby meeting various data push requirements and ensuring the security of data push.

[0077] Optionally, the data push rule includes a data filtering rule, a parameter mapping rule, and an encryption rule; in step S50, that is, the push data is obtained from the target data source according to the push requirement information, and the push data is pushed to the target push interface through the data push rule, including:

[0078] S501, determining a push time according to the push time rule; the push time includes a start push time and / or a receive push time;

[0079] S502, obtaining first push data matched with the push requirement information from the target data source according to the push time;

[0080] S503, filtering the first push data according to the data filtering rule to generate second push data;

[0081] S504, mapping and converting the second push data according to the parameter mapping rule to generate third push data;

[0082] S505, encrypt the third push data according to the encryption rule to generate fourth push data;

[0083] S506, push the fourth push data to the target push interface.

[0084] It can be understood that the push time includes the start push time of the data sender and the receiving push time of the data receiver. The push time rule is a rule for determining the start push time and the receiving push time, and the push time rule includes a real-time push rule and a timing push rule. In an embodiment, a real-time push and real-time receiving mode can be used, then the data sender directly obtains the push data matched with the push demand information from the target data source for sending, and the data receiver directly receives the push data through the target push interface. A timing push and timing receiving mode can be used, then the data sender obtains the push data matched with the push demand information according to the pre-set start push time for sending, and the data receiver receives the push data according to the pre-set receiving push time. A real-time push and timing receiving mode can also be used, then the data sender directly obtains the push data matched with the push demand information from the target data source for sending, and the data receiver receives the push data according to the pre-set receiving push time.

[0085] The first push data refers to the initial push data obtained from the target data source, the second push data refers to the push data after filtering and checking the initial push data according to the data filtering rule, the third push data refers to the push data after mapping and converting the second push data according to the parameter mapping rule, and the fourth push data refers to the push data after encrypting the third push data according to the encryption rule.

[0086] In an embodiment, when the data push task is started, the data source initializer of the server reads the data source parameter configuration information of the target data source in the data source list, creates a target data source instance and puts it into the data source pool. For example, when the target data source is a Kafka type data source, a Kafka consumer is created, when the target data source is a database type data source, a data query timing task is created, and when the target data source is a data file type data source, a directory file monitor is created. When the server reads the data source parameter configuration information, a uniquely determined data source configuration version number corresponding to the data source parameter configuration information is generated, and the data source configuration version number is recorded in the cache, and the created target data source instance carries the current data source configuration version number. When the data push task is started, the server also creates a data source refresh program executed by a separate thread, which is used to monitor the data source configuration version number in the cache. When the user modifies the data source configuration online, the data source configuration version number in the cache is updated synchronously, and when the data source refresh program detects a change in the data source configuration version number, it compares whether the data source configuration version number carried by the target data source instance in the data source pool is the latest one. If it is not the latest one, a new version of the target data source instance is created to replace the old version of the target data source instance, so as to realize the instant update of the data source. When the data push task is started, the server receives the first push data matching the push requirement information through the target data source instance.

[0087] When the target data source instance receives the first push data, it is handed over to the data processor in the server for processing, and the data processor includes a filter (filtering rule), a data converter (parameter mapping rule) and an encrypter (encryption rule). The data processor queries the data push task that subscribes to the target data source and is in an open state, and then triggers the data push process of the data push task. After the data processor processes the first push data according to the data filtering rule, the parameter mapping rule and the encryption rule, the fourth push data is obtained. Among them, the server assembles the filter chain according to the configuration order of the filter component, if there is a rule expression, the first push data and the expression text are transmitted to the rule engine for analysis, then the parsed first push data and the judgment value are compared, if the verification is passed, the second push data is generated and the subsequent processing is continued, otherwise the current data push task is ended. The server constructs a data converter according to the parameter mapping rule configuration information, fills the source data field value of the first push data or the first push data after the rule engine analysis into the push interface request parameter object, and performs non-empty verification on the fields configured in the push interface data structure. If the verification is passed, the third push data is generated and the subsequent processing is continued, otherwise the current data push task is ended. The server calls the selected target encrypter and encryption key to encrypt the third push data configured with encryption in the push interface data structure to generate the fourth push data.

[0088] After the server completes the data processing work through the filter, the data converter and the encryptor, the server pushes the processed fourth push data to the selected target push interface. The server creates a target push interface instance according to the type of the target push interface and the corresponding push interface configuration information, and pushes the fourth push data to the data receiver through the target push interface instance. When the target push interface instance is a Kafka type interface instance, a Kafka producer is first created according to the Kafka cluster address in the push interface configuration information, and then the fourth push data is written to the message topic. At the same time, the Kafka producer is cached in the memory, and the next time the Kafka producer is obtained, the Kafka producer will be tried to be obtained from the cache. When the target push interface instance is an http type interface instance, an http client is first created, and then the fourth push data is pushed to the data receiver according to the interface address and the message header in the push interface configuration information. When the target push interface instance is a database type interface instance, a database connection pool is first created according to the database address, the database name, the account and the password in the push interface configuration information, then the fourth push data is inserted into the placeholder in the sql statement, and finally the connection is obtained from the database connection pool to execute the sql statement, and the fourth push data is written to the database of the data receiver.

[0089] The server pre-establishes a push compensation list, and when the target push interface fails to receive the fourth push data, the fourth push data and the target push interface are saved to the push compensation list. The compensation push program can read the fourth push data from the push compensation list at regular intervals to automatically compensate and push until the target push interface successfully receives the fourth push data. The push compensation list can also be manually selected through the management interface of the data push platform for manual compensation. Whether the target push interface successfully receives the fourth push data or not, a data push record is generated and saved to the ES log, and the monitoring and statistical program will regularly count the data push records in the ES log on the same day, and generate a push statistical table according to the statistical results. The management interface of the data push platform queries the statistical results from the push statistical table and displays the page, so as to realize online real-time monitoring. At the same time, the monitoring and statistical program will generate an early warning email and send it to the user enterprise corresponding to the data push task if the statistical result is found to be abnormal and reaches the alarm value during the statistical process.

[0090] The data pushing method of the embodiment connects a data sender and a data receiver based on a form of a data pushing platform and provides a data pushing service, effectively solves the problems of pushing demand customization and management and maintenance difficulty through demand management, data pushing, monitoring and early warning and online maintenance, reduces the technical threshold of data pushing of a user enterprise, and enables a user enterprise without software development capability to also realize data pushing. When data receiving fails, corresponding pushing data can be collected and periodically pushed to avoid omission of the pushing data and ensure completeness of the pushing data.

[0091] The data pushing method of the embodiment connects a data sender and a data receiver based on a form of a data pushing platform and provides a data pushing service. The data pushing platform, as a general standard platform, can be open source and free for use by a user enterprise. The data pushing platform collects data pushing demands of different related departments or user cooperative enterprises. When a user enterprise needs to be connected with corresponding related departments or cooperative enterprises, the corresponding data pushing demands can be directly referenced to a local. The user enterprise splices pushing demand information such as demand information, expected completion time, application reason, relevant correspondence and interface of a person to be connected into an internal approval form for internal approval. If the approval is passed, a data pushing task can be directly created through the data pushing platform. The user enterprise can configure online data sources, pushing interfaces, data filtering modes, parameter mapping modes, encryption modes, pushing time and pushing frequency and the like, so as to create the data pushing task. The server of the data pushing platform quickly completes data pushing by executing the data pushing task, thereby improving data pushing efficiency.

[0092] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0093] In an embodiment, a data pushing device is provided, which corresponds to the data pushing method in the above embodiment. As shown in the figure, the data pushing device comprises a pushing demand receiving module 10, an approval result obtaining module 20, a configuration page generating module 30, a pushing configuration determining module 40 and a data pushing module 50. The functions of each module are described in detail as follows. Figure 2 The pushing demand receiving module 10 is configured to receive pushing demand information.

[0094] The pushing demand receiving module 10 is configured to receive pushing demand information.

[0095] The approval result obtaining module 20 is configured to obtain an approval result of the pushing demand information.

[0096] The configuration page generating module 30 is configured to generate a pushing configuration page according to the pushing demand information if the approval result is approval.

[0097] The push configuration determination module 40 is configured to determine a target data source, a target push interface and a data push rule through the push configuration page.

[0098] The data push module 50 is configured to acquire push data from the target data source according to the push demand information, and push the push data to the target push interface through the data push rule, so that a data receiver receives the push data through the target push interface.

[0099] Optionally, the push configuration determination module 40 comprises:

[0100] The data source list generation unit is configured to receive data source parameter configuration information of a plurality of data sources through the data source configuration page, and generate a data source list according to the data source parameter configuration information.

[0101] The target data source selection unit is configured to receive a data source selection instruction through the data source configuration page, and select the target data source from the data source list according to the data source selection instruction.

[0102] Optionally, the push configuration determination module 40 further comprises:

[0103] The push interface list generation unit is configured to receive push interface parameter configuration information of a plurality of push interfaces through the push interface configuration page, and generate a push interface list according to the push interface parameter configuration information.

[0104] The target push interface selection unit is configured to receive a push interface selection instruction through the push interface configuration page, and select the target push interface from the push interface list according to the push interface selection instruction.

[0105] Optionally, the push configuration determination module 40 further comprises:

[0106] The judgment information receiving unit is configured to receive data judgment condition information and judgment value information through the data filtering rule configuration page.

[0107] The data filtering rule generation unit is configured to generate the data filtering rule according to the data judgment condition information and the judgment value information.

[0108] Optionally, the push configuration determination module 40 further comprises:

[0109] The data source field information acquisition unit is configured to acquire data source field information of the target data source.

[0110] The push interface parameter information acquisition unit is configured to acquire push interface parameter information of the target push interface.

[0111] The parameter mapping rule generation unit is configured to receive mapping relationship information of the data source field information and the push interface parameter information through the parameter mapping rule configuration page, and generate the parameter mapping rule according to the mapping relationship information.

[0112] Optionally, the push configuration determination module 40 further includes:

[0113] The target encryptor selection unit is configured to receive an encryptor selection instruction through the encryption rule configuration page, and select a target encryptor from an encryptor list according to the encryptor selection instruction.

[0114] The encryption rule configuration unit is configured to receive encryption rule configuration information of the target encryptor through the encryption rule configuration page, and configure the encryption rule according to the encryption rule configuration information.

[0115] Optionally, the data push module 50 includes:

[0116] The push time determination unit is configured to determine a push time according to the push time rule; the push time includes a start push time and / or a receive push time.

[0117] The first push data acquisition unit is configured to acquire first push data matched with the push demand information from the target data source according to the push time.

[0118] The second push data generation unit is configured to filter the first push data according to the data filtering rule, and generate second push data.

[0119] The third push data generation unit is configured to perform mapping conversion on the second push data according to the parameter mapping rule, and generate third push data.

[0120] The fourth push data generation unit is configured to encrypt the third push data according to the encryption rule, and generate fourth push data.

[0121] The fourth push data push unit is configured to push the fourth push data to the target push interface.

[0122] The specific limitation of the data push device can refer to the limitation of the data push method in the above, which will not be repeated here. Each module in the above data push device can be realized by software, hardware and combination thereof in whole or in part. The above each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to call and execute the operation corresponding to each module by the processor.

[0123] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a readable storage medium and an internal memory. The readable storage medium stores an operating system, computer-readable instructions and a database. The internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The database of the computer device is used to store data involved in the data push method. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer-readable instructions are executed by the processor, a data push method is implemented. The readable storage medium provided in this embodiment includes a non-volatile readable storage medium and a volatile readable storage medium.

[0124] In one embodiment, a computer device is provided, comprising a memory, a processor, and computer-readable instructions stored in the memory and executable on the processor. When the processor executes the computer-readable instructions, the following steps are implemented:

[0125] Receive push demand information;

[0126] Obtaining the approval result of the push demand information;

[0127] If the approval result is approved, a push configuration page is generated according to the push requirement information;

[0128] Determine the target data source, target push interface and data push rules through the push configuration page;

[0129] The push data is acquired from the target data source according to the push demand information, and the push data is pushed to the target push interface according to the data push rule, so that a data recipient receives the push data through the target push interface.

[0130] In one embodiment, one or more computer-readable storage media storing computer-readable instructions are provided. The computer-readable storage media provided in this embodiment include non-volatile computer-readable storage media and volatile computer-readable storage media. The computer-readable instructions stored on the computer-readable storage media, when executed by one or more processors, implement the following steps:

[0131] Receive push demand information;

[0132] Obtaining the approval result of the push demand information;

[0133] If the approval result is approved, a push configuration page is generated according to the push requirement information;

[0134] determine a target data source, a target push interface and a data push rule through the push configuration page;

[0135] According to the push demand information, push data is acquired from the target data source, and the push data is pushed to the target push interface through the data push rule, so that the push data is received by a data receiver through the target push interface.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a non-volatile readable storage medium or a volatile readable storage medium. When the computer readable instructions are executed, the processes of the above-mentioned embodiments can be included. Any reference to memory, storage, database or other medium used in each embodiment provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0137] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0138] The above-described embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data push method, characterized by, The method comprises the following steps: receiving push demand information of a data receiver, and sending the push demand information to a data sender; the push demand information is description information about pushing data provided by the data receiver; obtaining an approval result of the data sender on the push demand information; if the approval result is approval, generating a push configuration page according to the push demand information; the push configuration page comprises a data source configuration page, a data filtering rule configuration page, a parameter mapping rule configuration page, an encryption rule configuration page and a push interface configuration page; determining a target data source, a target push interface and a data push rule through the push configuration page; obtaining push data from the target data source according to the push demand information, and pushing the push data to the target push interface through the data push rule, so that the data receiver receives the push data through the target push interface; wherein the data push rule comprises a data filtering rule; the method further comprises the following steps: receiving data judgment condition information and judgment value information through the data filtering rule configuration page; generating the data filtering rule according to the data judgment condition information and the judgment value information.

2. The data push method of claim 1, wherein, the method further comprises the following steps: receiving data source parameter configuration information of a plurality of data sources through the data source configuration page, and generating a data source list according to the data source parameter configuration information; receiving a data source selection instruction through the data source configuration page, and selecting the target data source from the data source list according to the data source selection instruction.

3. The data push method of claim 1, wherein, the method further comprises the following steps: receiving push interface parameter configuration information of a plurality of push interfaces through the push interface configuration page, and generating a push interface list according to the push interface parameter configuration information; receiving a push interface selection instruction through the push interface configuration page, and selecting the target push interface from the push interface list according to the push interface selection instruction.

4. The data push method of claim 1, wherein, the data push rule comprises a parameter mapping rule; the method further comprises the following steps: obtaining data source field information of the target data source; obtaining push interface parameter information of the target push interface; receiving mapping relationship information of the data source field information and the push interface parameter information through the parameter mapping rule configuration page, and generating the parameter mapping rule according to the mapping relationship information.

5. The data push method of claim 1, wherein, the data push rule comprises an encryption rule; the method further comprises the following steps: receiving an encryptor selection instruction through the encryption rule configuration page, and selecting a target encryptor from an encryptor list according to the encryptor selection instruction. The encryption rule configuration information of the target encryptor is received through the encryption rule configuration page, and the encryption rule is configured according to the encryption rule configuration information.

6. The data push method of claim 1, wherein, The data pushing rule includes a pushing time rule, a data filtering rule, a parameter mapping rule and an encryption rule. The data pushing method comprises the following steps: The pushing time includes a starting pushing time and / or a receiving pushing time; The first pushing data matched with the pushing demand information is acquired from the target data source according to the pushing time; The second pushing data is generated by filtering the first pushing data according to the data filtering rule; The third pushing data is generated by mapping and converting the second pushing data according to the parameter mapping rule; The fourth pushing data is generated by encrypting the third pushing data according to the encryption rule; The fourth pushing data is pushed to the target pushing interface.

7. A data push apparatus, characterized by comprising: The data pushing method comprises the following steps: The pushing demand receiving module is configured to receive pushing demand information of a data receiver and send the pushing demand information to a data sender. The pushing demand information is description information about pushing data provided by the data receiver. The approval result obtaining module is configured to obtain an approval result of the data sender on the pushing demand information. The configuration page generating module is configured to generate a pushing configuration page according to the pushing demand information if the approval result is approval. The pushing configuration determining module is configured to determine a target data source, a target pushing interface and a data pushing rule through the pushing configuration page. The data pushing module is configured to acquire pushing data from the target data source according to the pushing demand information and push the pushing data to the target pushing interface through the data pushing rule, so that the data receiver receives the pushing data through the target pushing interface. The pushing configuration determining module comprises: The judgment information receiving unit is configured to receive data judgment condition information and judgment value information through the data filtering rule configuration page. The data filtering rule generating unit is configured to generate the data filtering rule according to the data judgment condition information and the judgment value information.

8. A computer device comprising a memory, a processor, and computer readable instructions stored in the memory and executable on the processor, wherein, The processor executes the computer readable instructions to implement the data pushing method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored computer-readable instructions, wherein, The computer readable instructions are executed by one or more processors to enable the one or more processors to perform the data pushing method in any one of claims 1 to 6.

Images