Business data processing method, device and equipment

Abstract

The embodiment of the present invention provides a method, device and equipment for processing business data, the method comprising: obtaining first business data to be processed, determining a business keyword in the first business data, and determining whether the first business data has been counted according to the business keyword, and when it is determined that the first business data has not been counted, performing statistical processing on the first business data to obtain statistical results, and storing the statistical results in a first database. In the above data processing process, statistical processing is performed on the first business data only when it is determined that the first business data has not been counted, thereby achieving data idempotence at the business level, avoiding repeated statistical processing of repeated business data, and ensuring end-to-end data consistency.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of big data technology, and in particular, to a method, device and equipment for processing business data. Background Art

[0002] In some business scenarios, it is necessary to collect statistics and display the real-time accumulated values ​​of big data. Taking the e-commerce field as an example, the e-commerce platform needs to collect real-time statistics on data such as order quantity, order amount, and product sales, and display the statistical results to understand the sales situation in a timely manner.

[0003] In the prior art, business data is generated by various business systems. After the business systems generate business data, the business data needs to be transmitted and processed between multiple nodes, and the final processing results are submitted to the database for display on the large screen.

[0004] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art: In the above business data processing process, there may be end-to-end data inconsistency. For example, some orders may be counted repeatedly, resulting in the displayed order statistics being inconsistent with the order situation actually generated by the business system. Summary of the invention

[0005] The embodiments of the present invention provide a method, apparatus and device for processing business data, which can avoid repeated processing of repeated business data, thereby ensuring end-to-end data consistency.

[0006] In a first aspect, an embodiment of the present invention provides a method for processing business data, including:

[0007] Acquire first business data to be processed;

[0008] determining a service keyword in the first service data, and determining whether the first service data has been counted according to the service keyword;

[0009] When it is determined that the first service data has not been counted, performing statistical processing on the first service data to obtain a statistical result;

[0010] The statistical results are stored in the first database.

[0011] In a possible implementation manner, determining whether the first service data has been counted according to the service keyword includes:

[0012] Determine whether the service keyword exists in a second database, where the second database is used to store service keywords in the counted service data;

[0013] If the service keyword exists in the second database, determining that the first service data has been counted;

[0014] If the service keyword does not exist in the second database, it is determined that the first service data has not been counted.

[0015] In a possible implementation manner, determining the service keyword in the first service data includes:

[0016] Determining a service type corresponding to the first service data;

[0017] A service keyword in the first service data is determined according to the service type and a preset corresponding relationship, wherein the preset corresponding relationship is used to indicate a corresponding relationship between different service types and different service keywords.

[0018] In a possible implementation manner, before determining whether the first service data has been counted according to the service keyword, the method further includes:

[0019] According to the service type, obtaining a processing method corresponding to the service type, the processing method including a parsing method and / or a verification method;

[0020] According to the processing method, the first business data is parsed and / or verified.

[0021] In a possible implementation, the method is applied to a server using the distributed stream processing framework Flink, and the statistical results are stored in a first database, including:

[0022] The statistical results are stored in the first database through the Flink, and checkpoint information is generated according to the statistical results, and the checkpoint information is used to restore data according to the checkpoint information when the Flink fails.

[0023] In a possible implementation, storing the statistical result in the first database through the Flink, and generating checkpoint information according to the statistical result, includes:

[0024] Sending a pre-commit request to the Flink and the first database;

[0025] After receiving a pre-commit confirmation message from the Flink and receiving a pre-commit confirmation message from the first database, a commit request is sent to the Flink and the first database, so that the Flink stores the statistical results in the first database and generates checkpoint information according to the statistical results.

[0026] In a possible implementation, obtaining the first service data to be processed includes:

[0027] A first message is obtained from a Kafka message queue through the Flink, where the first message includes the first business data. The Kafka message queue is used to store messages generated from each business system server, and there are no duplicate messages in the Kafka message queue.

[0028] In a possible implementation manner, performing statistical processing on the first service data to obtain statistical results includes:

[0029] Acquire statistical frequency information, and add the first service data to a cache queue corresponding to the statistical frequency information;

[0030] According to the statistical frequency information, statistical processing is performed on the service data in the cache queue to obtain the statistical result.

[0031] In a second aspect, an embodiment of the present invention provides a device for processing business data, including:

[0032] An acquisition module, used for acquiring first business data to be processed;

[0033] a processing module, configured to determine a service keyword in the first service data, and determine whether the first service data has been counted according to the service keyword;

[0034] The processing module is further configured to perform statistical processing on the first business data to obtain a statistical result when it is determined that the first business data has not been counted;

[0035] The submission module is used to store the statistical results in the first database.

[0036] In a possible implementation, the processing module is specifically used to:

[0037] Determine whether the service keyword exists in a second database, where the second database is used to store service keywords in the counted service data;

[0038] If the service keyword exists in the second database, determining that the first service data has been counted;

[0039] If the service keyword does not exist in the second database, it is determined that the first service data has not been counted.

[0040] In a possible implementation, the processing module is specifically used to:

[0041] Determining a service type corresponding to the first service data;

[0042] A service keyword in the first service data is determined according to the service type and a preset corresponding relationship, wherein the preset corresponding relationship is used to indicate a corresponding relationship between different service types and different service keywords.

[0043] In a possible implementation, the processing module is further configured to:

[0044] According to the service type, obtaining a processing method corresponding to the service type, the processing method including a parsing method and / or a verification method;

[0045] According to the processing method, the first business data is parsed and / or verified.

[0046] In a possible implementation, the method is applied to a server using a distributed stream processing framework Flink, and the submission module is specifically used to:

[0047] The statistical results are stored in the first database through the Flink, and checkpoint information is generated according to the statistical results, and the checkpoint information is used to restore data according to the checkpoint information when the Flink fails.

[0048] In a possible implementation, the submission module is specifically used to:

[0049] Sending a pre-commit request to the Flink and the first database;

[0050] After receiving a pre-commit confirmation message from the Flink and receiving a pre-commit confirmation message from the first database, a commit request is sent to the Flink and the first database, so that the Flink stores the statistical results in the first database and generates checkpoint information according to the statistical results.

[0051] In a possible implementation, the acquisition module is specifically used to:

[0052] A first message is obtained from a Kafka message queue through the Flink, where the first message includes the first business data. The Kafka message queue is used to store messages generated from each business system server, and there are no duplicate messages in the Kafka message queue.

[0053] In a possible implementation, the processing module is specifically used to:

[0054] Acquire statistical frequency information, and add the first service data to a cache queue corresponding to the statistical frequency information;

[0055] According to the statistical frequency information, statistical processing is performed on the service data in the cache queue to obtain the statistical result.

[0056] In a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor;

[0057] The memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions to implement the method as described in any one of the first aspects.

[0058] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0059] The method, device and equipment for processing business data provided by the embodiment of the present invention include: obtaining the first business data to be processed, determining the business keyword in the first business data, and determining whether the first business data has been counted according to the business keyword, and when it is determined that the first business data has not been counted, statistical processing is performed on the first business data to obtain statistical results, and the statistical results are stored in a first database. In the above data processing process, statistical processing is performed on the first business data only when it is determined that the first business data has not been counted, thereby realizing data idempotence at the business level, avoiding repeated statistical processing of repeated business data, and ensuring end-to-end data consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0061] Figure 1 A schematic diagram of a possible system architecture applicable to an embodiment of the present invention;

[0062] Figure 2 A flowchart of a method for processing business data provided by an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of a processing process of first service data provided by an embodiment of the present invention;

[0064] Figure 4 A flowchart of a method for processing business data provided by another embodiment of the present invention;

[0065] Figure 5 A schematic diagram of a two-phase submission provided by an embodiment of the present invention;

[0066] Figure 6 A schematic diagram of the Flink fault tolerance mechanism provided by an embodiment of the present invention;

[0067] Figure 7 A schematic diagram of a component architecture provided for an embodiment of the present invention;

[0068] Figure 8 A schematic diagram of a system for processing business data provided by an embodiment of the present invention;

[0069] Fig. 9 A schematic diagram of the structure of a device for processing business data provided by one embodiment of the present invention;

[0070] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0072] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0073] First, the concepts involved in the embodiments of the present invention are introduced.

[0074] Idempotence: This means that the impact of executing a function multiple times is the same as the impact of executing it only once.

[0075] Kafka: Kafka is a distributed, publish / subscribe-based messaging system developed and open-sourced by LinkedIn. It is widely used due to its distributed nature and high throughput.

[0076] Flink: is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data flow engine written in Java and Scala. Flink executes in a data-parallel and pipelined manner.

[0077] Exactly once: For a message, the receiving end ensures that it is received only once. Exactly-once semantics ensures that there is no data loss or duplication in the final data processing result and data ingestion.

[0078] The embodiment of the present invention is applicable to business scenarios in the field of big data where real-time statistics of business data are required. Figure 1 A possible system architecture of an embodiment of the present invention is introduced. Figure 1 FIG. 1 is a schematic diagram of a possible system architecture applicable to an embodiment of the present invention. Figure 1 As shown, the system architecture includes a business system server, a message publishing and subscription server, a big data processing server, and a database.

[0079] Among them, each business system server serves as the source of service-side business data, that is, the business system server can obtain the business data generated by the user through the client operating the business system. Each business system server is connected to the message publishing and subscription server. Each business system server sends the generated business data to the message publishing and subscription server. The big data processing server is also connected to the message publishing and subscription server, and the big data processing server can subscribe to business data from the message subscription server.

[0080] For the message publishing and subscription server, each business system server is the producer of business data, and the big data processing server is the consumer of business data. In this way, the business data generated by each business system server can be transmitted to the big data processing server in real time, so that the big data processing server can perform real-time statistical calculations on the business data and obtain statistical results.

[0081] Furthermore, the big data processing server submits the statistical results to the database. The data in the database can be displayed on a large screen. Through the above process, real-time statistics and display of business data are achieved.

[0082] Combination Figure 1,Since the message publishing and subscription servers and the big data processing servers need to receive and process the business data generated by multiple business systems, ,the message publishing and subscription servers and the big data processing servers usually adopt a ,distributed cluster service architecture.

[0083] In the prior art, after the business system generates business data, the business data needs to be transmitted and processed between multiple nodes, and finally submitted to the database for display on the large screen. However, in the above-mentioned business data processing process, since the business data needs to be transmitted and processed between multiple nodes, when there are problems such as network failure, node failure, program failure, etc., it often leads to repeated sending of messages between nodes. For example, after the sender sends a certain business data, it may not receive the confirmation from the receiver in time due to network delays and other reasons. In order to avoid losing business data, the sender may send the business data repeatedly. For another example, if a node fails, after the failure is restored, some business data before the failure may be repeatedly processed to try to ensure that business data is not lost.

[0084] It can be seen that in the above business data processing process, it is inevitable that business data is sent or processed repeatedly, resulting in end-to-end data inconsistency. For example, taking the order business as an example, when some orders are counted repeatedly, the final displayed order statistics will be inconsistent with the actual order situation generated by the business system.

[0085] The method for processing business data provided by the embodiment of the present invention can avoid repeated processing of business data and ensure the consistency of end-to-end data.

[0086] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0087] Figure 2 A flowchart of a method for processing business data provided by an embodiment of the present invention is shown below. Figure 1 The big data processing server in is executed. Taking the processing process of the first business data as an example, Figure 3 FIG. 1 is a schematic diagram of a processing process of first service data provided by an embodiment of the present invention. Figure 2 and Figure 3 As shown, the method of this embodiment includes:

[0088] S201: Acquire first business data to be processed.

[0089] The first business data may be any business data received by the big data processing server.

[0090] Optionally, after obtaining the first business data, the first business data may be preprocessed. The preprocessing includes, but is not limited to, parsing processing, verification processing, etc. The parsing processing is used to parse the business data to obtain information of each field in the business data. The verification processing is used to verify information of each field in the business data, for example, performing non-empty verification on certain fields, performing value range verification on certain fields, etc.

[0091] In actual application scenarios, business data of various business types may be included. Different preprocessing methods may be used for business data of different business types. In a possible implementation, Figure 3 As shown, the processing method corresponding to the business type can be obtained according to the business type corresponding to the first business data, and the processing method includes a parsing method and / or a verification method; according to the processing method, the first business data is parsed and / or verified.

[0092] Optionally, there may be an intersection of preprocessing methods corresponding to different business types. In order to facilitate the reuse of preprocessing methods, in this embodiment, the preprocessing requirements of each business scenario can be summarized in advance to determine the basic preprocessing method and the extended preprocessing method. For the basic preprocessing method, a basic component is provided, and for the extended preprocessing method, an extensible component is provided. Then, a correspondence between the business type and the basic component and the extensible component is established. In this way, it is only necessary to determine which components need to be used to preprocess the business data based on the business type, thereby improving the reusability and extensibility of the preprocessing method.

[0093] S202: Determine a service keyword in the first service data, and determine whether the first service data has been counted according to the service keyword.

[0094] In order to avoid the problem of end-to-end data inconsistency caused by repeated transmission or repeated processing of business data, the embodiment of the present invention provides a business idempotent solution to ensure that business data is only counted once from the business level.

[0095] Specifically, each business data includes a business keyword, which is a keyword in the business data that can uniquely identify the business data. The business keyword may include one or more keywords. For example, in the order business, the order number can be used as a business keyword. In the real-time order sales business, the order number and order status fields can be used as business keywords.

[0096] Optionally, the business keywords corresponding to the business data of different business types may be different. For example, for the order business type, the order number may be used as the business keyword; for the user business type, the user ID may be used as the business keyword; etc. Therefore, the business keywords corresponding to different business types may be determined in advance, and the corresponding relationship between the business types and the business keywords may be established.

[0097] In a possible implementation manner, the service keyword in the first service data may be determined in the following feasible manner: determining the service type corresponding to the first service data; and determining the service keyword in the first service according to the service type and the above correspondence.

[0098] In this embodiment, after determining the service keyword in the first service data, it can be determined whether the first service data has been counted according to the service keyword, that is, whether the first service data is repeated service data. If the first service data has been counted, the first service data can be discarded to avoid repeated counting of repeated service data. If the first service data has not been counted, the subsequent S203 can be continued.

[0099] In the specific implementation process, the service keywords of the service data that have been counted can be recorded. In this way, when processing the first service data, it can be determined whether the first service data has been counted according to the previously recorded service keywords.

[0100] In a possible implementation, a second database may be used to store business keywords in the business data that has been counted. Accordingly, S202 may specifically include: determining whether the business keyword in the first business data exists in the second database. If so, determining that the first business data has been counted. If not, determining that the first business data has not been counted.

[0101] When it is determined that the first business data has not been counted, the business keyword in the first business data can be stored in the second database. In this way, when new business data including the same business keyword is subsequently received, the new business data can be determined as duplicate data according to the second database, thereby achieving the purpose of deduplication of business data.

[0102] Optionally, the second database may use a Remote Dictionary Server (Redis) database. Redis database is an open source log-type, Key-Value database written in ANSI C language, supporting the network, and can be based on memory or persistent. By using the Redis database to store the business keywords of the processed business data, the processing efficiency of deduplication of business data can be improved.

[0103] In this embodiment, by determining whether the first business data has been counted according to the business keyword in the first business data, idempotent data processing at the business level is achieved, ensuring that repeated business data is only processed once. Taking the order business as an example, after the idempotent processing at the business level of this embodiment, it can be ensured that the order type of the same order can only be accumulated once, and the cancellation of an order can only be deducted once.

[0104] It should be noted that the idempotent processing at the business level in this embodiment is not equivalent to the idempotent processing at the message level. After the idempotent processing at the message level, it can only be guaranteed that there are no duplicate messages. However, it is possible that two different messages carry the same business data. Therefore, there may still be a problem of repeated statistics of business data. This embodiment can essentially avoid repeated statistics of business data and ensure end-to-end data consistency by performing idempotent processing at the business level.

[0105] S203: When it is determined that the first service data has not been counted, statistical processing is performed on the first service data to obtain a statistical result.

[0106] S204: storing the statistical results in a first database.

[0107] In this embodiment, statistical processing is performed on the first business data only when it is determined that the first business data has not been counted, thereby avoiding repeated statistics on the business data and ensuring the accuracy of the statistical results.

[0108] Optionally, when the business data is counted in this embodiment, statistics can be performed according to a preset statistical frequency. For example, statistics can be performed once every 5 minutes, once every 15 minutes, once every half an hour, once every 1 hour, once every 12 hours, etc. The statistical frequency information can be pre-configured according to the actual application scenario.

[0109] Optionally, different business types may correspond to different statistical frequency information.

[0110] In a possible implementation, S203 may specifically include: obtaining statistical frequency information, and adding the first business data to a cache queue corresponding to the statistical frequency information. According to the statistical frequency information, statistical processing is performed on the business data in the cache queue to obtain statistical results. Taking the statistical frequency of 15 minutes as an example, a 15-minute time window is set, and a cache queue is set for the time window. When it is determined that the first business data has not been counted, the first business data is first added to the cache queue. When the 15-minute time window expires, statistical processing is performed on the business data in the cache queue to obtain statistical results.

[0111] After obtaining the statistical results, the statistical results are submitted to the first database. The data in the first database can be displayed on a large screen.

[0112] Optionally, the first database in this embodiment may adopt a relational database, for example, a MySQL database, to facilitate visual display of data.

[0113] The method for processing business data provided in this embodiment includes: obtaining first business data to be processed, determining a business keyword in the first business data, and determining whether the first business data has been counted according to the business keyword, and when it is determined that the first business data has not been counted, performing statistical processing on the first business data to obtain statistical results, and storing the statistical results in a first database. In the above data processing process, statistical processing is performed on the first business data only when it is determined that the first business data has not been counted, thereby achieving data idempotence at the business level, avoiding repeated statistical processing of repeated business data, and ensuring end-to-end data consistency.

[0114] Based on any of the above embodiments, the technical solution of the present invention is described in more detail below in conjunction with a specific embodiment.

[0115] Figure 4 A flowchart of a method for processing business data provided by another embodiment of the present invention. The method of this embodiment can be applied to Figure 1 As shown in the system architecture, in this system architecture, the message publishing and subscription server can use the Kafka processing framework, the big data processing server uses the distributed stream processing framework Flink, and the database can use the Mysql database.

[0116] like Figure 4 As shown, the method of this embodiment may include:

[0117] S401: Obtain a first message from a Kafka message queue through Flink, where the first message includes the first business data. The Kafka message queue is used to store messages generated by each business system server, and there are no duplicate messages in the Kafka message queue.

[0118] In this embodiment, the business data generated by each business system server is connected to Kafka. Kafka is a high-throughput distributed publish-subscribe message system, which is generally used as a message access source in the big data field. Kafka can receive messages sent by each business system server and maintain the messages in the message queue. Flink can subscribe to messages from Kafka and obtain messages from the Kafka message queue to obtain the business data in the message.

[0119] Among them, Kafka uses version 0.11.0.0 or above. Kafka version 0.11.0.0 introduces the message idempotence function. For messages sent repeatedly by the business system server, Kafka will only write them into the message queue once. In the specific implementation, it is necessary to enable the Exactly Once semantics function for Kafka.

[0120] In this embodiment, by adopting the exactly-once semantic function of Kafka, there are no duplicate messages in the Kafka message queue, which ensures idempotence at the message level and ensures data consistency from Kafka to Flink.

[0121] S402: Determine a service keyword in the first service data, and determine whether the first service data has been counted according to the service keyword.

[0122] S403: When it is determined that the first service data has not been counted, statistical processing is performed on the first service data to obtain a statistical result.

[0123] It should be understood that the specific implementation of S402 and S403 is different from Figure 2 S202 and S203 in the example are similar and will not be described here. S402 and S403 can be implemented in Flink, which ensures the data consistency within Flink at the business level.

[0124] S404: The statistical results are stored in a MySQL database through Flink, and checkpoint information is generated according to the statistical results. The checkpoint information is used to restore data according to the checkpoint information when Flink fails.

[0125] Specifically, you can configure the statistical frequency information in Flink according to the actual application scenario. Then, you can perform statistics on business data based on the statistical frequency information configured in Flink, and store the statistical results in the MySQL database through Flink.

[0126] In actual applications, when Flink submits data to the MySQL database, if Flink fails or crashes and restarts, in order to ensure that business data is not lost, Flink may send data to the MySQL database again after it restarts. However, the data may have been received by the MySQL database, resulting in repeated data submission.

[0127] In this embodiment, in order to avoid the problem of Flink repeatedly submitting data to the MySQL database, a two-phase commit method is adopted. Among them, Flink uses version 1.4.0 and above, and the MySQL database uses version 5.5 and above. Both versions of Flink and MySQL database support two-phase commit. In this embodiment, by combining the two-phase commit of Flink and MySQL database, the problem of repeated data submission can be avoided.

[0128] In one possible implementation, Figure 5 A schematic diagram of a two-phase commit provided by an embodiment of the present invention. Figure 5 As shown, a pre-commit request is sent to Flink and the Mysql database; after receiving the pre-commit confirmation message from Flink and the pre-commit confirmation message from the Mysql database, a commit request is sent to Flink and the Mysql database, so that Flink stores the statistical results in the Mysql database and generates checkpoint information according to the statistical results.

[0129] The two-phase submission of this embodiment includes: a pre-submission phase and a formal submission phase. In the pre-submission phase, the big data processing server acts as a coordinator and sends a pre-submission request to Flink and the MySQL database. If the pre-submission confirmation message of Flink is received, and the pre-submission confirmation message of the MySQL database is received, it means that both Flink and the MySQL database have met the data submission conditions. Then, the formal submission phase is entered, and a submission request is sent to Flink and the MySQL database. Flink will submit the statistical results to the MySQL database. At the same time, Flink will also generate checkpoint information based on the statistical results. By adopting a two-phase submission, the data consistency from Flink to the MySQL database can be guaranteed.

[0130] Furthermore, Flink in this embodiment also supports a fault-tolerant mechanism, which is implemented by continuously creating snapshots of distributed data streams and their state consistency. Figure 6 A schematic diagram of the Flink fault tolerance mechanism provided by an embodiment of the present invention. Figure 6As shown, turn on Flink's checkpoint function. During data processing, Flink can periodically take snapshots of the data processed by each operator and generate checkpoint information. In other words, Flink will back up the data at time intervals. When Flink encounters a failure (such as device failure, network failure, software failure, etc.), Flink stops the distributed data flow. Flink restarts all operators and resets each operator based on the most recent checkpoint information to restore it to the state of the most recent backup. It then re-executes from the state of the most recent checkpoint.

[0131] In this embodiment, by combining Flink's two-phase commit and checkpoint functions, it is ensured that checkpoint information is generated while submitting data to the MySQL database. In this way, it is ensured that the data for which checkpoints are generated are all submitted, and the data for which checkpoints are not generated are all unsubmitted. When Flink needs to recover data due to a failure, the data is recovered and execution is restarted according to the checkpoint information, which can avoid submitting duplicate data to the MySQL database and ensure the data consistency from Flink to the MySQL database.

[0132] In a possible implementation manner, this embodiment may be implemented using basic components and differentiated configurations. Figure 7 Schematic diagram of component architecture provided by an embodiment of the present invention. Figure 7 As shown, the common content of different business scenarios can be implemented using basic components. For example, basic components may include: business consumption components, business parsing and verification components, business idempotent components, Flink computing components, data submission components, Flink fault tolerance and fault recovery components, etc. Each component can follow the SPI design principle to facilitate users to perform a small amount of personalized coding. The same business scenario can be directly selected and configured according to the business model provided by this solution, that is, only a small amount of differentiated configuration is required to use it. Among them, differentiated configuration may include: business type configuration, Kafka configuration, Redis configuration, Flink configuration, Mysql configuration, etc.

[0133] Figure 8 Schematic diagram of a system for processing business data provided by an embodiment of the present invention. Figure 8As shown in the figure, in the data source stage, Kafka versions 0.11.0.0 and above are used to connect to various business systems to achieve idempotent processing at the message level. In the data processing stage, Flink versions 1.4.0 and above are used to implement parsing and verification according to business types to ensure reusability; Redis database is used to implement idempotent processing at the business level to avoid repeated statistics of business data; checkpoint function is supported to ensure data consistency in the event of a failure; two-phase commit is supported to avoid repeated submission of data. In the data display stage, Mysql database versions 5.5 and above are used to implement and cooperate with Flink's two-phase commit.

[0134] In this embodiment, through the above data processing process, the data consistency of the three stages of data source (Kafka), data processing (Flink), and data display (Mysql) is guaranteed, and end-to-end data consistency is achieved.

[0135] Fig. 9 The structure diagram of a device for processing business data provided by an embodiment of the present invention is shown in FIG. The device may be in the form of software and / or hardware. Fig. 9 As shown, the service data processing device 10 provided in this embodiment includes: an acquisition module 11, a processing module 12 and a submission module 13. Among them, the acquisition module 11 is used to acquire the first service data to be processed;

[0136] A processing module 12, configured to determine a service keyword in the first service data, and determine whether the first service data has been counted according to the service keyword;

[0137] The processing module 12 is further configured to perform statistical processing on the first business data to obtain statistical results when it is determined that the first business data has not been counted;

[0138] The submission module 13 is used to store the statistical results in the first database.

[0139] In a possible implementation, the processing module 12 is specifically configured to:

[0140] Determine whether the service keyword exists in a second database, where the second database is used to store service keywords in the counted service data;

[0141] If the service keyword exists in the second database, determining that the first service data has been counted;

[0142] If the service keyword does not exist in the second database, it is determined that the first service data has not been counted.

[0143] In a possible implementation, the processing module 12 is specifically configured to:

[0144] Determining a service type corresponding to the first service data;

[0145] A service keyword in the first service data is determined according to the service type and a preset corresponding relationship, wherein the preset corresponding relationship is used to indicate a corresponding relationship between different service types and different service keywords.

[0146] In a possible implementation, the processing module 12 is further configured to:

[0147] According to the service type, obtaining a processing method corresponding to the service type, the processing method including a parsing method and / or a verification method;

[0148] According to the processing method, the first business data is parsed and / or verified.

[0149] In a possible implementation, the method is applied to a server using the distributed stream processing framework Flink, and the submission module 13 is specifically used for:

[0150] The statistical results are stored in the first database through the Flink, and checkpoint information is generated according to the statistical results, and the checkpoint information is used to restore data according to the checkpoint information when the Flink fails.

[0151] In a possible implementation, the submitting module 13 is specifically used for:

[0152] Sending a pre-commit request to the Flink and the first database;

[0153] After receiving a pre-commit confirmation message from the Flink and receiving a pre-commit confirmation message from the first database, a commit request is sent to the Flink and the first database, so that the Flink stores the statistical results in the first database and generates checkpoint information according to the statistical results.

[0154] In a possible implementation, the acquisition module 11 is specifically used for:

[0155] A first message is obtained from a Kafka message queue through the Flink, where the first message includes the first business data. The Kafka message queue is used to store messages generated from each business system server, and there are no duplicate messages in the Kafka message queue.

[0156] In a possible implementation, the processing module 12 is specifically configured to:

[0157] Acquire statistical frequency information, and add the first service data to a cache queue corresponding to the statistical frequency information;

[0158] According to the statistical frequency information, statistical processing is performed on the service data in the cache queue to obtain the statistical result.

[0159] The business data processing device provided in this embodiment can be used to execute the technical solution of any of the above method embodiments. Its implementation principles and technical effects are similar and will not be described in detail here.

[0160] Fig.10 The structure diagram of an electronic device provided by an embodiment of the present invention. The electronic device can be used as a big data processing server. Fig.10 As shown, the electronic device 20 of this embodiment includes: a processor 21 and a memory 22; wherein the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program stored in the memory to implement the method for processing business data in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0161] Optionally, the memory 22 may be independent or integrated with the processor 21 .

[0162] When the memory 22 is a device independent of the processor 21 , the electronic device 20 may further include: a bus 23 for connecting the memory 22 and the processor 21 .

[0163] Optionally, the electronic device 20 may further include a communication component 24 for communicating with other devices.

[0164] The electronic device provided in this embodiment can be used to execute the technical solution in any of the above method embodiments. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0165] An embodiment of the present invention further provides a computer-readable storage medium. When instructions in the storage medium are executed by a processor, the method for processing business data in any of the above method embodiments can be executed.

[0166] An embodiment of the present invention further provides a chip, comprising: a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and the processor runs the computer program to execute the technical solution in any of the above method embodiments.

[0167] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0168] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0170] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (English: processor) to perform some steps of the method described in each embodiment of the present invention.

[0171] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0172] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk or an optical disk, etc.

[0173] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present invention is not limited to only one bus or one type of bus.

[0174] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0175] An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0176] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing business data, characterized in that: include: Obtain a first message from a Kafka message queue through Flink, where the first message includes first business data. The Kafka message queue is used to store messages generated from each business system server, and an exactly-once semantic function is enabled for the Kafka message queue so that there are no duplicate messages in the Kafka message queue. According to the business type corresponding to the first business data, obtaining a processing method corresponding to the business type, wherein the processing method includes a parsing method and / or a verification method; according to the processing method, performing parsing processing and / or verification processing on the first business data; wherein the processing method is implemented based on a basic preprocessing component and an extended preprocessing component; determining a business keyword in the first business data, and determining whether the first business data has been counted according to the business keyword; When it is determined that the first service data has not been counted, performing statistical processing on the first service data to obtain a statistical result; Storing the statistical results in a first database; The method is applied to a server using the distributed stream processing framework Flink, and stores the statistical results in a first database, including: Sending a pre-commit request to the Flink and the first database; After receiving a pre-commit confirmation message from the Flink and receiving a pre-commit confirmation message from the first database, a commit request is sent to the Flink and the first database, so that the Flink stores the statistical results in the first database and generates checkpoint information according to the statistical results, and the checkpoint information is used to restore data according to the checkpoint information when the Flink fails.

2. The method according to claim 1, characterized in that: Determining whether the first service data has been counted according to the service keyword includes: Determine whether the service keyword exists in a second database, where the second database is used to store service keywords in the counted service data; If the service keyword exists in the second database, determining that the first service data has been counted; If the service keyword does not exist in the second database, it is determined that the first service data has not been counted.

3. The method according to claim 2, characterized in that Determining the service keyword in the first service data includes: Determining a service type corresponding to the first service data; A service keyword in the first service data is determined according to the service type and a preset corresponding relationship, wherein the preset corresponding relationship is used to indicate a corresponding relationship between different service types and different service keywords.

4. The method according to any one of claims 1 to 3, characterized in that: Performing statistical processing on the first service data to obtain statistical results includes: Acquire statistical frequency information, and add the first service data to a cache queue corresponding to the statistical frequency information; According to the statistical frequency information, statistical processing is performed on the service data in the cache queue to obtain the statistical result.

5. A device for processing business data, characterized in that: include: An acquisition module is used to acquire a first message from a Kafka message queue through Flink, where the first message includes first business data, and the Kafka message queue is used to store messages generated from each business system server, and enable an exactly-once semantic function for the Kafka message queue so that there are no duplicate messages in the Kafka message queue; A processing module, used to determine a service keyword in the first service data, and determine whether the first service data has been counted according to the service keyword; The processing module is further configured to perform statistical processing on the first business data to obtain a statistical result when it is determined that the first business data has not been counted; A submission module, used for storing the statistical results in a first database; The commit module is specifically configured to send a pre-commit request to the Flink and the first database; after receiving a pre-commit confirmation message from the Flink and a pre-commit confirmation message from the first database, send a commit request to the Flink and the first database, so that the Flink stores the statistical result in the first database, and generates checkpoint information according to the statistical result, and the checkpoint information is used to restore data according to the checkpoint information when the Flink fails; The processing module is specifically used to obtain a processing method corresponding to the business type according to the business type corresponding to the first business data, and the processing method includes a parsing method and / or a verification method; according to the processing method, the first business data is parsed and / or verified; wherein the processing method is implemented based on a basic preprocessing component and an extended preprocessing component.

6. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor executes the computer-executable instructions to implement the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 4 when executed by a processor.

8. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.

Images