Data processing method and device based on flink and related equipment

Abstract

The invention provides a data processing method and device based on flink and related equipment, and relates to the technical field of computers and the Internet. The method comprises the following steps: intercepting a request message sent by an flink service framework; determining that the request message is sent to an external database by the flink service framework to request the external data from the external database, wherein the request message is generated when the flink service framework receives the service data and determines to request the external data from the external database according to a service rule corresponding to the service data; sequentially storing the request messages in a sending waiting queue; and when the number of the request messages in the sending waiting queue is greater than or equal to a first threshold value, or the waiting time of the sending waiting queue is greater than or equal to a second threshold value, sending the request messages in the sending waiting queue to an external database in batches so as to request external data from the external database in batches. According to the embodiment of the invention, the business data in the flink can be supplemented through the external data.

Description

Technical Field

[0001] The present application relates to the field of computer and Internet technology, and in particular to a flink-based data processing method, device, electronic device, and computer-readable storage medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] Real-time data reports in current business scenarios are usually constructed by processing wide tables through the flink business framework. The basic steps are as follows: first, the flink business framework subscribes to the Binlog data (binary log files) of multiple business tables in the database Mysql (an open source cross-platform database management system based on SQL queries). After processing, the Binlog data is connected to the flink task framework, and then the flink task framework associates multiple business table data together to build wide table data, thereby creating real-time data reports corresponding to the business.

[0004] However, when a real-time data report of a business is generated through the Flink business framework, the data in the real-time data report is usually incomplete due to missing data. Summary of the invention

[0005] The purpose of this application is to provide a data processing method, device, electronic device and computer-readable storage medium based on Flink, which can supplement the business data in the Flink business framework with external data to generate real-time data reports.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] An embodiment of the present application provides a flink-based data processing method, including: intercepting a request message sent by a flink business framework; determining that the request message is sent by the flink business framework to an external database to request external data from the external database, wherein the request message is generated when the flink business framework receives business data and determines to request external data from the external database according to a business rule corresponding to the business data; storing the request messages in a sending waiting queue in sequence; when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or the waiting time of the sending waiting queue is greater than or equal to a second threshold, sending the request messages in the sending waiting queue in batches to the external database to request external data in batches from the external database.

[0008] In some embodiments, the external database includes a first external database and a second external database, and the sending waiting queue includes a first sending waiting queue and a second sending waiting queue, the first sending waiting queue is used to cache request messages sent to the first external database, and the second sending waiting queue is used to cache request messages sent to the second external database; wherein, the request messages are stored in the sending waiting queues in sequence, including: if it is determined that the request message is sent to the first external database, the request message is stored in the first sending waiting queue; if it is determined that the request message is sent to the second external database, the request message is stored in the second sending waiting queue.

[0009] In some embodiments, when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or the waiting time of the sending waiting queue is greater than or equal to a second threshold, the request messages in the sending waiting queue are sent in batches to the external database to batch request external data from the external database, including: when the number of request messages in the first sending waiting queue is greater than or equal to the first threshold, or the waiting time of the first sending waiting queue is greater than or equal to a second threshold, the request messages in the first sending waiting queue are sent in batches to the first external database to batch request external data from the first external database.

[0010] In some embodiments, the method also includes: intercepting the external request result issued by the external database; determining that the external request result is the external request result sent by the external database to the flink in response to the request message; storing the external request results in a receiving waiting queue in sequence; when the number of external request results in the receiving waiting queue is greater than or equal to a third threshold, or the waiting time of the receiving waiting queue is greater than or equal to a fourth threshold, sending the request messages in the receiving waiting queue in batches to the flink business framework so that the flink business framework performs business processing on the external request results.

[0011] An embodiment of the present application provides a flink-based data processing method, which is applied to a flink business framework, including: acquiring business data; determining business rules corresponding to the business data; generating a request message according to the business rules; sending the request message so as to request external data from an external database through the request message, wherein the request message is intercepted by an intermediate component and stored in a sending waiting queue, wherein the intermediate component will send the intimacy messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or when the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to batch request external data from the external database; receiving an external request result corresponding to the request message; performing business processing on the business data and the external request result returned by the external database according to the business rules.

[0012] In some embodiments, obtaining business data includes: subscribing to data in a first table and a second table in a production library; when it is determined that the data in the first table and the second table have changed, obtaining the first table data in the first table and the second table data in the second table, wherein the first table data and the second table data are the business data; wherein the method further includes: assembling the first table data and the second table data according to the business rules to generate wide table data; wherein generating a request message according to the business rules includes: determining that the wide table data needs to be supplemented with data according to the business rules; generating the request message according to the business rules, so as to request the external data from the external database according to the request message, so as to supplement the wide table data with the external data.

[0013] An embodiment of the present application provides a flink-based data processing device, which is deployed on an intermediate component and includes: a request message interception module, a service nature determination module, a storage module and a batch request module.

[0014] Among them, the request message interception module is used to intercept the request message issued by the flink business framework; the business nature determination module can be used to determine that the request message is sent by the flink business framework to an external database to request external data from the external database, wherein the request message is generated when the flink business framework receives business data and determines to request external data from the external database according to the business rules corresponding to the business data; the storage module can be used to store the request messages in the sending waiting queue in sequence; the batch request module can be used to send the request messages in the sending waiting queue in batches to the external database when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to batch request external data from the external database.

[0015] An embodiment of the present application provides a flink-based data processing device, which is deployed on a flink business framework and includes: a business data acquisition module, a business rule acquisition module, a request message generation module, a request message sending module, a request result receiving module, and a request result receiving module.

[0016] Among them, the business data acquisition module is used to acquire business data; the business rule acquisition module is used to determine the business rules corresponding to the business data; the request message generation module is used to generate a request message according to the business rules; the request message sending module is used to send the request message so as to request external data from an external database through the request message, wherein the request message is intercepted by the intermediate component and stored in a sending waiting queue, wherein the intermediate component will send the intimacy messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or when the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to batch request external data from the external database; the request result receiving module is used to receive the external request result corresponding to the request message; the business processing module is used to perform business processing on the business data and the external request result returned by the external database according to the business rules.

[0017] An embodiment of the present application proposes an electronic device, which includes: a memory and a processor; the memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement any of the above-mentioned Flink-based data processing methods.

[0018] An embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored to implement the Flink-based data processing method as described in any one of the above items.

[0019] The embodiment of the present application proposes a computer program product or a computer program, which includes computer program instructions, and the computer program instructions are stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and the processor executes the computer program instructions to implement the above-mentioned flink-based data processing method.

[0020] The flink-based data processing method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application, on the one hand, the flink business framework can determine the corresponding business rules based on the received business data, and then determine whether to request external data based on the business rules, and send a request message when it is determined that the external data needs to be requested to request external data to supplement the business data in the flink business framework; on the other hand, the intermediate component will intercept a single request message issued by the flink business framework and temporarily store the single request message and send it to the external database in batches after a certain number or a certain period of time has been temporarily stored, so as to request data in batches from the external database, thereby avoiding the problem that the external database is continuously attacked by multiple requests, resulting in a slow response speed of the external database request.

[0021] It should be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A scenario schematic diagram of a Flink-based data processing method or a Flink-based data processing device that can be applied to an embodiment of the present application is shown.

[0024] Figure 2 It is a schematic diagram of a framework of a data processing method based on Flink according to an exemplary embodiment.

[0025] Figure 3 The diagram is a schematic diagram of a single synchronization query method according to an exemplary embodiment.

[0026] Figure 4 The diagram is a schematic diagram of a single asynchronous query method according to an exemplary embodiment.

[0027] Figure 5 The figure is a flowchart of a data processing method based on Flink according to an exemplary embodiment.

[0028] Figure 6 It is a schematic diagram of a framework of a data processing method based on Flink according to an exemplary embodiment.

[0029] Figure 7 The figure is a flowchart of a method for storing a request message according to an exemplary embodiment.

[0030] Figure 8 The figure is a flowchart of a method for receiving an external request result according to an exemplary embodiment.

[0031] Fig. 9 A flink-based data processing method is shown according to an exemplary embodiment.

[0032] Fig.10 A method for acquiring business data is shown according to an exemplary embodiment.

[0033] Fig.11 It is a framework diagram of a data processing method based on Flink according to an exemplary embodiment.

[0034] Fig.12 It is a block diagram of a Flink-based data processing device according to an exemplary embodiment.

[0035] Fig.13 It is a block diagram of a Flink-based data processing device according to an exemplary embodiment.

[0036] Fig.14 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application is shown. DETAILED DESCRIPTION

[0037] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this application will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0038] Those skilled in the art know that the implementation of the present application can be a system, device, equipment, method or computer program product. Therefore, the present application can be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0039] Features, structures or characteristics described in the present application may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical solution of the present application may be put into practice and one or more of the specific details may be omitted, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present application.

[0040] The accompanying drawings are only schematic diagrams of the present application. The same reference numerals in the drawings represent the same or similar parts, and thus their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0042] In the description of the present application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a kind of association relationship describing the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different; the terms "comprising", "including" and "having" are used to express the meaning of open inclusion and mean that in addition to the listed elements / components / etc., there may be other elements / components / etc.

[0043] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0044] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0045] First, some terms involved in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0046] Flink: An open source real-time data processing framework that facilitates calculation of various data dimensions or data assembly.

[0047] Mysql: A database used to store data.

[0048] Binlog: Logs generated when data changes occur in MySQL.

[0049] Wide table data: A data table assembled by splicing multiple MySQL data tables according to business rules. The assembled data table usually has many columns, so it is usually called a wide table, and the data stored in it is called wide table data.

[0050] External data: In addition to the business data in MySQL, other data required for processing wide table data. External data may come from a system or a database. Business data in MySQL can refer to data generated in real time during the business process, such as order data generated in real time in the commodity purchase business. External data can refer to data generated in non-real time, such as some warehouse data, commodity identification data, etc.

[0051] Synchronous: When a call is issued, it will not return until the result is obtained; once the call returns, the return value is obtained. In other words, the caller actively waits for the result of the call.

[0052] Asynchronous: Just the opposite of synchronous, after the call is issued, the call returns directly, so no result is returned. When an asynchronous process call is issued, the caller will not get the result immediately. Instead, after the call is issued, the callee can respond to the result through status, notification or callback function.

[0053] The foregoing introduces some terminology concepts involved in the embodiments of the present application. The following introduces the technical features involved in the embodiments of the present application.

[0054] The exemplary implementation scheme of the present application is described in detail below with reference to the accompanying drawings.

[0055] Figure 1A scenario schematic diagram of a Flink-based data processing method or a Flink-based data processing device that can be applied to an embodiment of the present application is shown.

[0056] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present application.

[0057] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0058] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual reality devices, smart homes, etc.

[0059] The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device.

[0060] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms, etc. This application does not impose any restrictions on this.

[0061] The server 105 may, for example, intercept a request message sent by the flink business framework; the server 105 may, for example, determine that the request message is sent by the flink business framework to an external database to request external data from the external database, wherein the request message is generated when the flink business framework receives business data and determines to request external data from the external database according to business rules corresponding to the business data; the server 105 may, for example, store the request messages in a sending waiting queue in sequence; the server 105 may, for example, send the request messages in the sending waiting queue in batches to the external database when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to batch request external data from the external database.

[0062] It should be understood that Figure 1 The number of terminal devices, networks and servers in the figure is only for illustration. The server 105 may be a physical server or may be composed of multiple servers. According to actual needs, it may have any number of terminal devices, networks and servers.

[0063] Under the above system architecture, an embodiment of the present application provides a flink-based data processing method, which can be executed by any electronic device with computing and processing capabilities.

[0064] In some embodiments, real-time data reports can be constructed by processing wide table models through Flink. Figure 2 The construction of real-time data reports can include the following steps: first, the Flink business framework subscribes to multiple different business tables (such as Figure 2 The Binlog data of Table A and Table B in the table is processed and connected to the Flink task. Then, Flink uses business rules to associate multiple table data together to build wide table data (such as Figure 2 As shown, the messages in table A can be associated with the messages in table B to generate wide table data), thereby creating a real-time data report.

[0065] Among them, the MySQL database may store real-time business data, such as order data, video data, audio data, etc., and this application does not impose any restrictions on this.

[0066] However, in some scenarios, some data columns in the wide table model data need to be supplemented by querying other data sources. For example, when generating a real-time data table for a customer's order, when obtaining the product name in the business data, it may be necessary to continue to obtain the library name of the library where the product is located. The library name may not exist in the business data, but in an external database. In this case, the business data in Flink needs to be supplemented by external data in the external database.

[0067] In some embodiments, the flink business framework obtains business data by subscribing to the business data, and then determines the business rules corresponding to the business data. When it is determined according to the business rules that the business data needs to be supplemented by external data, the external data is supplemented.

[0068] In some embodiments, the Flink business framework can supplement external data through the following two solutions: single synchronous query and single asynchronous query.

[0069] Among them, a single synchronous query solution processing flow may include the following: Figure 3 Steps shown:

[0070] 1. Initiate a request to synchronously call the external data source for message 1;

[0071] 2. Then synchronously wait for the external data source to return data;

[0072] 3. After receiving the data returned by the external data source of message 1, the transaction processing of message 1 is completed;

[0073] 4. Then continue to initiate a request to synchronously call the external data source for message 2;

[0074] 5. It also synchronously waits for the external data source to return data;

[0075] 6. After receiving the data returned by the synchronous call to the external data source by message 2, the transaction processing of message 2 is completed.

[0076] Among them, a single asynchronous query solution processing flow may include the following: Figure 4 Steps shown:

[0077] 1. First, initiate an asynchronous call to the external data source for message 1;

[0078] 2. Unlike the synchronous query solution, the asynchronous query solution does not synchronously wait for the external data to return results, but continues to initiate asynchronous call requests for external data of message 2, message 3, and message 4;

[0079] 3. After the message request is sent, Flink will not wait for the request result synchronously, but continue to process subsequent messages such as messages 5, 6, 7, etc.;

[0080] 4. When the external system message 1 request is processed, the request client will call back the flink framework to return the request result of message 1; 5. flink completes the transaction of message 1. Similarly, messages 2, 3, and 4 are processed.

[0081] Through the above method, the flink business framework can determine the corresponding business rules according to the received business data, and then determine whether to request external data according to the business rules, and send a request message when it is determined that external data needs to be requested to request external data to supplement the business data in the flink business framework, so that the rules in the flink business framework can be adjusted.

[0082] Therefore, the above method can complete the supplement of external data in a simple and convenient way.

[0083] However, comparing the two solutions: if it takes 1 second to wait for the result of an external access request, it takes 4 seconds to process 4 messages using the synchronous query solution, and it only takes 1 second to process 4 messages using the asynchronous query solution, saving 3 seconds. Therefore, the throughput of a single asynchronous query solution is higher than that of a single synchronous query solution.

[0084] A single synchronous query solution needs to wait for the external system to return the result before processing subsequent messages after processing each message. Flink tasks usually process tens of thousands, hundreds of thousands, or even millions of messages per minute, so a large amount of time is spent on synchronous waiting in the synchronous query solution, which is very inefficient.

[0085] Although the efficiency of a single asynchronous query solution is greatly improved compared to a single synchronous query solution, Flink tasks usually process tens of thousands, hundreds of thousands, or even millions of messages per minute. When processing a large number of messages, the mechanism of processing only one message at a time will cause two problems.

[0086] 1. Poor performance: Accessing external data with messages at the level of tens of thousands, hundreds of thousands, or millions requires a lot of network request time.

[0087] 2. It will cause great pressure on the external data source system: This is artificially created system pressure, which may cause the external data system to crash and be unable to provide services.

[0088] In order to solve the above technical problems, this application also provides the following flink-based data processing method.

[0089] Figure 5is a flowchart of a data processing method based on flink according to an exemplary embodiment. The method provided in the embodiment of the present application can be completed by an intermediate component, which can be deployed on any electronic device with computing and processing capabilities, such as the above Figure 1 The server or terminal device in the embodiment may also be a combination of a server and a terminal device. In the following embodiments, the intermediate component is deployed on the server as an example for illustration, but the present application is not limited to this.

[0090] Reference Figure 5 The flink-based data processing method provided in the embodiment of the present application may include the following steps.

[0091] Step S502: intercept the request message sent by the flink business framework.

[0092] like Figure 6 As shown, the Flink business framework can first subscribe to multiple different business tables in MySQL (an open source cross-platform database management system based on SQL queries) (such as Figure 6 The Binlog data of Table A and Table B in the table is processed and connected to the flink task. Then, flink uses business rules to associate multiple table data together to build wide table data (such as Figure 6 As shown, the messages in table A can be associated with the messages in table B to generate wide table data), thereby creating a real-time data report; according to the rules corresponding to the real-time report, it is determined that external data needs to be supplemented when constructing the real-time data report.

[0093] In some embodiments, when it is determined through business rules that the real-time data report needs to be supplemented with external data, the flink business framework will send a request message to request the external data from the external database.

[0094] In some embodiments, Flink is used as a real-time business framework, which can only send one message at a time.

[0095] In some embodiments, the intermediate component may intercept each request message sent by the flink business framework.

[0096] Step S504, determining that the request message is sent by the flink business framework to the external database to request external data from the external database, wherein the request message is generated when the flink business framework receives the business data and determines to request the external data from the external database according to the business rules corresponding to the business data.

[0097] In some embodiments, after the intermediate component intercepts the request message sent by the flink business framework, it determines whether the request message is sent by the flink business framework to an external database to request external data from the external database.

[0098] If the request message is sent by the Flink business framework to an external database to request external data from the external database, it will further determine which external database the request message is sent to, and then determine the sending waiting queue corresponding to the external database.

[0099] In some embodiments, the intermediate component sets different sending waiting queues for different external databases. In other words, one external database corresponds to one sending waiting queue.

[0100] Step S506: Store the request messages in a sending waiting queue in sequence.

[0101] In some embodiments, after the intermediate component determines the sending waiting queue corresponding to the request message, it will store the request message in the sending waiting queue in sequence.

[0102] In general, the intermediate component will continuously intercept the request messages sent by the Flink business framework, and then store the intercepted request messages requesting data from the external database in sequence in the sending waiting queue corresponding to the external database.

[0103] Step S508, when the number of request messages in the sending waiting queue is greater than or equal to the first threshold, or the waiting time of the sending waiting queue is greater than or equal to the second threshold, the request messages in the sending waiting queue are sent in batches to the external database to batch request external data from the external database.

[0104] In some embodiments, when the number of request messages in a sending queue of an intermediate component is greater than or equal to a first threshold, or the waiting time of a sending waiting queue is greater than or equal to a second threshold, the request messages in the sending waiting queue are packaged and sent to an external database corresponding to the sending waiting queue to request external data in batches from the external database.

[0105] Through the above method, on the one hand, the flink business framework can determine the corresponding business rules based on the received business data, and then determine whether to request external data based on the business rules, and send a request message when it is determined that external data needs to be requested to request external data to supplement the business data in the flink business framework; on the other hand, the intermediate component will intercept a single request message issued by the flink business framework and temporarily store the single request message and send it to the external database in batches after a certain number or a certain period of time, so as to request data in batches from the external database, thereby avoiding the problem that the external database is continuously attacked by multiple requests and the response speed of the external database request is slow.

[0106] Figure 7 The figure is a flowchart of a method for storing a request message according to an exemplary embodiment.

[0107] In some embodiments, the external database may include a first external database and a second external database, and the sending waiting queue may include a first sending waiting queue and a second sending waiting queue, wherein the first sending waiting queue can be used to cache request messages sent to the first external database, and the second sending waiting queue can be used to cache request messages sent to the second external database.

[0108] refer to Figure 7 In the illustrated embodiment, the request message storage method may include the following steps.

[0109] Step S702: intercept the request message sent by the flink business framework.

[0110] Step S704, determining that the request message is sent by the flink business framework to the external database to request external data from the external database, wherein the request message is generated when the flink business framework receives the business data and determines to request the external data from the external database according to the business rules corresponding to the business data.

[0111] Step S706: If it is determined that the request message is sent to the first external database, the request message is stored in a first sending waiting queue.

[0112] Step S708: If it is determined that the request message is sent to the second external database, the request message is stored in a second sending waiting queue.

[0113] Step S710, when the number of request messages in the first sending waiting queue is greater than or equal to the first threshold, or the waiting time of the first sending waiting queue is greater than or equal to the second threshold, the request messages in the first sending waiting queue are sent in batches to the first external database to batch request external data from the first external database.

[0114] Step S712, when the number of request messages in the second sending waiting queue is greater than or equal to the first threshold, or the waiting time of the second sending waiting queue is greater than or equal to the second threshold, the request messages in the second sending waiting queue are sent in batches to the second external database to batch request external data from the second external database.

[0115] Through the above method, request messages corresponding to different external databases can be stored in different sending waiting queues, which can achieve better diversion so that the same business can be processed together, avoiding problems such as the external database being continuously attacked by multiple requests, resulting in slow response speed of external database requests.

[0116] Figure 8 The figure is a flowchart of a method for receiving an external request result according to an exemplary embodiment.

[0117] refer to Figure 8 The above external request result receiving method may include the following steps.

[0118] Step S802, intercepting the external request result sent by the external database.

[0119] In some embodiments, the intermediate component may also intercept each message sent to the flink business framework.

[0120] Step S804, determining the external request result is that the external database is the external request result sent to Flink in response to the request message.

[0121] In some embodiments, if it is determined that the message sent to the flink business framework is an external request result sent by the external database to flink in response to the request message, step S806 may be executed.

[0122] Step S806, storing the external request results in the receiving waiting queue in sequence.

[0123] Step S808, when the number of external request results in the receiving waiting queue is greater than or equal to the third threshold, or the waiting time of the receiving waiting queue is greater than or equal to the fourth threshold, the request messages in the receiving waiting queue are sent in batches to the flink business framework so that the flink business framework performs business processing on the external request results.

[0124] In some embodiments, if the external database frequently responds to the flink business framework, it will also impact the flink business framework, causing the flink business framework to crash or slow down the processing speed. In order to solve the above problems, the present application also intercepts the external request results sent to the flink business framework through an intermediate component and stores them in a receiving waiting queue. Then, if the number of external request results in the receiving waiting queue is greater than or equal to the third threshold, or the waiting time of the receiving waiting queue is greater than or equal to the fourth threshold, the request messages in the receiving waiting queue are sent to the flink business framework in batches, so that the flink business framework can perform business processing on the external request results. In short, the business pressure of the filnk business framework can be alleviated by the above method.

[0125] Fig. 9 A flink-based data processing method is shown according to an exemplary embodiment.

[0126] In some embodiments, the flink-based data processing method can be applied to the flink business framework.

[0127] refer to Fig. 9 , the above-mentioned flink-based data processing method may include the following steps.

[0128] Step S902, obtaining business data.

[0129] like Figure 6 As shown, the Flink business framework can first subscribe to multiple different business tables in MySQL (an open source cross-platform database management system based on SQL queries) (such as Figure 6 The Binlog data of Table A and Table B in the table is processed and connected to the flink task. Then, flink uses business rules to associate multiple table data together to build wide table data (such as Figure 6 As shown, the messages in table A can be associated with the messages in table B to generate wide table data), thereby creating a real-time data report; according to the rules corresponding to the real-time report, it is determined that external data needs to be supplemented when constructing the real-time data report.

[0130] Step S904: determine the business rules corresponding to the business data.

[0131] Step S906: Generate a request message according to the business rules.

[0132] In some embodiments, when it is determined through business rules that the real-time data report needs to be supplemented with external data, the flink business framework will send a request message to request the external data from the external database.

[0133] In some embodiments, Flink is used as a real-time business framework, which can only send one message at a time.

[0134] Step S908, sending a request message to request external data from an external database through the request message, wherein the request message is intercepted by the intermediate component and stored in a sending waiting queue, wherein the intermediate component will send the intimacy messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold or when the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to request external data from the external database in batches.

[0135] Step S910: receiving an external request result corresponding to the request message.

[0136] In some embodiments, the external request results received by the flink business framework may be results returned one by one by an external database, or may be results returned in batches after being processed by an intermediate component. This application does not impose any restrictions on this.

[0137] Step S912: Perform business processing on the business data and the external request result returned by the external database according to the business rules.

[0138] In some embodiments, after receiving the external request result, the flink business framework can perform business processing on the business data and the external request result returned by the external database according to the business rules. This application does not limit the specific business processing content.

[0139] Fig.10 A method for acquiring business data is shown according to an exemplary embodiment.

[0140] refer to Fig.10 The above-mentioned business data acquisition method may include the following steps.

[0141] Step S1002, subscribing to data in the first table and the second table in the production library.

[0142] In some embodiments, the production database may refer to a MySQL database, and the data in the production database may be business data generated in real time.

[0143] In some embodiments, the production library may include a first table and a second table, the data corresponding to the first table may be a first type of data, and the data corresponding to the second table may be a second type of data. The first type of data may be order data for a first commodity, and the second type of data may be order data for a second commodity, and the present application is limited to this step.

[0144] Step S1004, when it is determined that the data in the first table and the second table have changed, the first table data in the first table and the second table data in the second table are obtained, wherein the first table data and the second table data are business data.

[0145] In some embodiments, when it is determined that data in the first table and the second table have changed, the first table data that have changed in the first table and the second table data that have changed in the second table are obtained, wherein the first table data and the second table data are business data.

[0146] Step S1006: assemble the first table data and the second table data according to business rules to generate wide table data.

[0147] Step S1008: determine the business rules corresponding to the business data.

[0148] Step S1010: Determine, based on business rules, whether the wide table data needs to be supplemented.

[0149] Step S1012: Generate a request message according to business rules.

[0150] Step S1014, sending a request message to request external data from an external database through the request message, wherein the request message is intercepted by the intermediate component and stored in a sending waiting queue, wherein the intermediate component will send the intimacy messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold or when the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to request external data from the external database in batches.

[0151] Step S1016, receiving the external request result corresponding to the request message.

[0152] Step S1018: Perform business processing on the business data and the external request results returned by the external database according to the business rules

[0153] Step S1020: Generate a request message according to the business rules, so as to request external data from the external database according to the request message, so as to supplement the wide table data with the external data.

[0154] Fig.11 It is a framework diagram of a data processing method based on Flink according to an exemplary embodiment.

[0155] refer to Fig.11 As shown in the framework diagram, the above-mentioned flink-based data processing method may include the following steps.

[0156] 1. First, Flink subscribes to the Binlog data of Table A and Table B in the production library.

[0157] 2. According to business rules, associate the data in Table A and Table B to assemble and generate wide table data.

[0158] 3. If the generated wide table data needs to be supplemented with external data, perform the external data supplement operation.

[0159] a) Development based on the "Batch Asynchronous Query Component (BatchRichAsyncFunction)" of the present application: external data service calls business logic.

[0160] b) Development based on the device "Batch Asynchronous Query Component (BatchRichAsyncFunction)" of the present invention: external data service callback interface business logic.

[0161] If the processing is successful: call back the completion interface (complete) of the "Batch Asynchronous Query Component (BatchRichAsyncFunction)".

[0162] If the processing fails: call back the exception handling interface (completeExceptionally) of the "Batch Asynchronous Query Component (BatchRichAsyncFunction)".

[0163] The wide table data stream is further processed and stored in the database or sent as a message via MQ.

[0164] The detailed principle of the batch asynchronous query component (BatchRichAsyncFunction) of this application device.

[0165] 1. If Fig.11 As shown, firstly, a request for asynchronously calling an external data source is initiated for message 1.

[0166] 2. The batch asynchronous query solution of the present invention is the same as the single asynchronous query technical solution in that when message 1 is received, subsequent message processing will not be blocked, but messages 2, 3, and 4 will continue to be initiated. However, unlike the single asynchronous query technical solution, the device of the present invention will not initiate an external data source request call, but convert messages 1, 2, 3, and 4 into request events, and add the request events to the internal waiting queue of the batch asynchronous query component (BatchRichAsyncFunction).

[0167] 3. After the message request is sent, Flink will not wait for the request result synchronously, but continue to process subsequent messages such as messages 5, 6, 7, etc.

[0168] 4. When the number of request events in the internal waiting queue reaches a certain number, or a certain time has passed since the first event was added to the queue, four messages will be used to initiate a batch asynchronous external data source access request.

[0169] 5. When the external system completes processing of the batch asynchronous requests, the request client will call back the Flink framework to return the request results of messages 1, 2, 3, and 4.

[0170] a) If the result is normal, the completion interface (complete) of BatchRichAsyncFunction is called back, and then Flink completes the transaction of the batch of messages.

[0171] b) If an exception occurs in the request, the exception handling interface (completeExceptionally) is used to customize the business exception.

[0172] In this embodiment, based on the single asynchronous operator RichAsyncFunction of Flink itself, the BatchRichAsyncFunction operator is developed by introducing a waiting queue to batch messages inside Flink. The key technical point is to introduce a waiting queue to optimize the asynchronous external data access once for each message processed into a batch asynchronous external data solution for processing multiple messages only once, which greatly improves the efficiency of data supplementation and reduces the pressure on the called system.

[0173] It should be noted that the various steps in the various embodiments of the above-mentioned data processing method based on flink can be mutually crossed, replaced, added, or deleted. Therefore, these reasonable permutations, combinations, and transformations of the data processing method based on flink should also fall within the scope of protection of this application, and the scope of protection of this application should not be limited to the embodiments.

[0174] Based on the same inventive concept, the present application also provides a flink-based data processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0175] Fig.12 is a block diagram of a data processing device based on Flink according to an exemplary embodiment. Fig.12 The flink-based data processing device 1200 provided in an embodiment of the present application can be deployed on a central component, including: a request message interception module 1201, a service nature determination module 1202, a storage module 1203 and a batch request module 1204.

[0176] Among them, the request message interception module 1201 can be used to intercept the request message sent by the flink business framework; the business nature determination module 1202 can be used to determine that the request message is sent by the flink business framework to an external database to request external data from the external database, wherein the request message is generated when the flink business framework receives the business data and determines to request external data from the external database according to the business rules corresponding to the business data; the storage module 1203 can be used to store the request messages in the sending waiting queue in sequence; the batch request module 1204 can be used to send the request messages in the sending waiting queue in batches to the external database when the number of request messages in the sending waiting queue is greater than or equal to the first threshold, or the waiting time of the sending waiting queue is greater than or equal to the second threshold, so as to batch request external data from the external database.

[0177] It should be noted that the request message interception module 1201, the service nature determination module 1202, the storage module 1203, and the batch request module 1204 correspond to S502 to S508 in the method embodiment, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content applied for in the above method embodiment. It should be noted that the above modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0178] In some embodiments, the external database includes a first external database and a second external database, the sending waiting queue includes a first sending waiting queue and a second sending waiting queue, the first sending waiting queue is used to cache request messages sent to the first external database, and the second sending waiting queue is used to cache request messages sent to the second external database; wherein, the storage module 1203 may include: a first waiting queue storage submodule and a second waiting queue storage submodule.

[0179] Among them, the first waiting queue storage submodule can be used to store the request message in the first sending waiting queue if it is determined that the request message is sent to the first external database; the second waiting queue storage submodule can be used to store the request message in the second sending waiting queue if it is determined that the request message is sent to the second external database.

[0180] In some embodiments, the batch request module 1204 may include: a first external database request module.

[0181] Among them, the first external database request module can be used to send the request messages in the first sending waiting queue to the first external database in batches when the number of request messages in the first sending waiting queue is greater than or equal to the first threshold, or the waiting time of the first sending waiting queue is greater than or equal to the second threshold, so as to batch request external data from the first external database.

[0182] In some embodiments, the data processing device 1200 may further include: a result interception module, an external request result determination module, a receiving waiting queue storage module, and a business processing module.

[0183] Among them, the result interception module can be used to intercept the external request result issued by the external database; the external request result determination module can be used to determine that the external request result is the external request result sent by the external database to flink in response to the request message; the receiving waiting queue storage module can be used to store the external request results in the receiving waiting queue in sequence; the business processing module can be used to send the request messages in the receiving waiting queue in batches to the flink business framework when the number of external request results in the receiving waiting queue is greater than or equal to the third threshold, or the waiting time of the receiving waiting queue is greater than or equal to the fourth threshold, so that the flink business framework can perform business processing on the external request results.

[0184] Since the functions of the device 1200 have been described in detail in the corresponding method embodiments, this application will not repeat them here.

[0185] Fig.13 is a block diagram of a data processing device based on Flink according to an exemplary embodiment. Fig.13 The flink-based data processing device 1300 provided in an embodiment of the present application can be deployed on a flink business framework, including: a business data acquisition module 1301, a business rule acquisition module 1302, a request message generation module 1303, a request message sending module 1304, a request result receiving module 1305 and a business processing module 1306.

[0186] Among them, the business data acquisition module 1301 can be used to acquire business data; the business rule acquisition module 1302 is used to determine the business rules corresponding to the business data; the request message generation module 1303 can be used to generate a request message according to the business rules; the request message sending module 1304 can be used to send a request message so as to request external data from an external database through a request message, wherein the request message is intercepted by the intermediate component and stored in a sending waiting queue, wherein the intermediate component will send the intimacy messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or when the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to batch request external data from the external database; the request result receiving module 1305 can be used to receive an external request result corresponding to the request message; the business processing module 1306 can be used to perform business processing on the business data and the external request results returned by the external database according to the business rules.

[0187] In some embodiments, the business data acquisition module 1301 may include: a subscription submodule and a table data acquisition submodule.

[0188] The subscription submodule can be used to subscribe to the data in the first table and the second table in the production library; the table data acquisition submodule can be used to acquire the first table data in the first table and the second table data in the second table when it is determined that the data in the first table and the second table have changed, wherein the first table data and the second table data are business data;

[0189] In some embodiments, the data processing device 1300 may further include: a wide table data assembly module.

[0190] The wide table data assembly module may be used to assemble the first table data and the second table data according to business rules to generate wide table data.

[0191] In some embodiments, the request message generation module 1303 may include: a service determination submodule and a supplementation submodule.

[0192] Among them, the business determination submodule can be used to determine that the wide table data needs to be supplemented according to business rules; the supplement submodule can be used to generate a request message according to business rules, so as to request external data from an external database according to the request message, so as to supplement the wide table data with external data.

[0193] Since the functions of the device 1300 have been described in detail in the corresponding method embodiments, this application will not repeat them here.

[0194] The modules and / or submodules involved in the embodiments described in the present application may be implemented by software or hardware. The modules and / or submodules described may also be arranged in a processor. The names of these modules and / or submodules do not, in certain cases, constitute limitations on the modules and / or submodules themselves.

[0195] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a part of a module or program segment, and a part of the above-mentioned module or program segment contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer program instructions.

[0196] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0197] Fig.14 The structure diagram of the electronic device suitable for implementing the embodiment of the present application is shown. It should be noted that: Fig.14 The electronic device 1400 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0198] like Fig.14 As shown, the electronic device 1400 includes a central processing unit (CPU) 1401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1402 or a program loaded from a storage part 1408 into a random access memory (RAM) 1403. In the RAM 1403, various programs and data required for the operation of the electronic device 1400 are also stored. The CPU 1401, the ROM 1402, and the RAM 1403 are connected to each other via a bus 1404. An input / output (I / O) interface 1405 is also connected to the bus 1404.

[0199] The following components are connected to the I / O interface 1405: an input section 1406 including a keyboard, a mouse, etc.; an output section 14012 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1408 including a hard disk, etc.; and a communication section 1409 including a network interface card such as a LAN card, a modem, etc. The communication section 1409 performs communication processing via a network such as the Internet. A drive 1410 is also connected to the I / O interface 1405 as needed. A removable medium 1412, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1410 as needed, so that a computer program read therefrom is installed into the storage section 1408 as needed.

[0200] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes computer program instructions for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1409, and / or installed from the removable medium 1412. When the computer program is executed by the central processing unit (CPU) 1401, the above-mentioned functions defined in the system of the present application are executed.

[0201] It should be noted that the computer-readable storage medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable computer program instructions. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The computer program instructions contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0202] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by a device, the device can realize functions including: intercepting a request message sent by the flink business framework; determining that the request message is sent by the flink business framework to an external database to request external data from the external database, wherein the request message is generated when the flink business framework receives the business data and determines to request external data from the external database according to the business rules corresponding to the business data; storing the request messages in a sending waiting queue in sequence; when the number of request messages in the sending waiting queue is greater than or equal to the first threshold, or the waiting time of the sending waiting queue is greater than or equal to the second threshold, the request messages in the sending waiting queue are sent in batches to the external database to batch request external data from the external database.

[0203] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer program instructions, the computer program instructions being stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0204] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution of the embodiment of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of computer program instructions for enabling an electronic device (which can be a server or a terminal device, etc.) to execute the method according to the embodiment of the present application.

[0205] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the application applied for herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not applied for in the present application. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0206] It should be understood that the present application is not limited to the detailed structures, drawings or implementations shown herein, but rather, the present application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data processing method based on flink, characterized in that: Applies to intermediate components, including: Intercept the request message sent by the flink business framework; Determine that the request message is sent by the flink business framework to an external database to request external data from the external database, wherein the request message is generated when the flink business framework receives business data and determines to request external data from the external database according to a business rule corresponding to the business data; The request messages are sequentially stored in a sending waiting queue; When the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or the waiting time of the sending waiting queue is greater than or equal to a second threshold, the request messages in the sending waiting queue are sent in batches to the external database to batch request external data from the external database.

2. The method according to claim 1, characterized in that: The external database includes a first external database and a second external database, the sending waiting queue includes a first sending waiting queue and a second sending waiting queue, the first sending waiting queue is used to cache request messages sent to the first external database, and the second sending waiting queue is used to cache request messages sent to the second external database; wherein the request messages are sequentially stored in the sending waiting queues, including: If it is determined that the request message is sent to the first external database, storing the request message in the first sending waiting queue; If it is determined that the request message is sent to the second external database, the request message is stored in the second sending waiting queue.

3. The method according to claim 2, characterized in that: When the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or the waiting time of the sending waiting queue is greater than or equal to a second threshold, sending the request messages in the sending waiting queue to the external database in batches to request external data in batches from the external database, including: When the number of request messages in the first sending waiting queue is greater than or equal to the first threshold, or the waiting time of the first sending waiting queue is greater than or equal to the second threshold, the request messages in the first sending waiting queue are sent in batches to the first external database to batch request external data from the first external database.

4. The method according to claim 1, characterized in that: The method further comprises: Intercepting the external request result sent by the external database; Determine that the external request result is an external request result sent by the external database to the flink in response to the request message; The external request results are sequentially stored in a receiving waiting queue; When the number of external request results in the receiving waiting queue is greater than or equal to the third threshold, or the waiting time of the receiving waiting queue is greater than or equal to the fourth threshold, the request messages in the receiving waiting queue are sent in batches to the flink business framework so that the flink business framework performs business processing on the external request results.

5. A data processing method based on flink, characterized in that: Applied to the Flink business framework, including: Get business data; Determining business rules corresponding to the business data; Generate a request message according to the business rules; The request message is sent out so as to request external data from an external database through the request message, wherein the request message is intercepted by an intermediate component and stored in a sending waiting queue, wherein the intermediate component sends the intimacy messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold or when the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to request external data from the external database in batches; Receive an external request result corresponding to the request message; Business processing is performed on the business data and the external request result returned by the external database according to the business rules.

6. The method according to claim 5, characterized in that: Obtain business data, including: Subscribe to the data in the first and second tables in the production database; When it is determined that the data in the first table and the second table have changed, obtaining the first table data in the first table and the second table data in the second table, wherein the first table data and the second table data are the business data; Wherein, the method further comprises: Assembling the first table data and the second table data according to the business rules to generate wide table data; Wherein, generating a request message according to the business rule includes: Determining, according to the business rules, that the wide table data requires data to be supplemented; The request message is generated according to the business rule, so as to request the external data from the external database according to the request message, so as to supplement the wide table data with the external data.

7. A data processing device based on flink, characterized in that: The intermediate components include: The request message interception module is used to intercept the request message sent by the flink business framework; A business nature determination module, used to determine that the request message is sent by the flink business framework to an external database to request external data from the external database, wherein the request message is generated when the flink business framework receives business data and determines to request external data from the external database according to a business rule corresponding to the business data; A storage module, used for sequentially storing the request messages in a sending waiting queue; The batch request module is used to send the request messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to batch request external data from the external database.

8. A data processing device based on flink, characterized in that: Deployed on the Flink business framework, including: Business data acquisition module, used to acquire business data; A business rule acquisition module, used to determine the business rules corresponding to the business data; A request message generating module, used to generate a request message according to the business rules; A request message sending module, used for sending the request message so as to request external data from an external database through the request message, wherein the request message is intercepted by an intermediate component and stored in a sending waiting queue, wherein the intermediate component will send the intimacy messages in the sending waiting queue to the external database in batches when the number of request messages in the sending waiting queue is greater than or equal to a first threshold, or when the waiting time of the sending waiting queue is greater than or equal to a second threshold, so as to batch request external data from the external database; A request result receiving module, used to receive the external request result corresponding to the request message; The business processing module is used to perform business processing on the business data and the external request result returned by the external database according to the business rules.

9. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the Flink-based data processing method according to any one of claims 1 to 6.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the flink-based data processing method according to any one of claims 1 to 6 is implemented.

Images