Data processing method and device and related product

Abstract

The invention provides a data processing method and device and a related product, and relates to the technical field of computers and interconnection. The data processing method comprises the steps that a data lake obtains target business data produced by a data producer and a global identification code of the target business data; the data lake pushes the target business data and the global identification code of the target business data to a data channel in the data lake for storage; and the data lake pushes the global identification code of the target business data to a verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to check whether the data in the data channel is abnormal or not. According to the embodiment of the invention, the accuracy of the data synchronized from the data warehouse to the data lake can be effectively audited.

Description

Technical Field

[0001] The present disclosure relates to the field of computer and Internet technology, and in particular to a data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the disclosure that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] With the development of big data technology, more and more companies will build data lakes and lake warehouse integrated architectures. In these scenarios, real-time data synchronization and consistency verification are key links. The data formats and storage media of source business systems vary. Traditional data synchronization tools have limited applicability in heterogeneous systems and cannot meet the needs of high-concurrency, low-latency real-time data collection.

[0004] In addition, existing solutions lack an effective auditing mechanism to ensure the accuracy of data synchronization, and the application of various new databases has brought new challenges to real-time data synchronization.

[0005] To this end, a data synchronization solution based on consistency verification is needed. Summary of the invention

[0006] The purpose of the present disclosure is to provide a data processing method, device, electronic device, computer-readable storage medium and computer program product, which ensure the accuracy and consistency of data after entering the lake through a multi-channel data transmission mechanism and automated auditing.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] The present disclosure provides a data processing method, including: a data lake acquires target business data produced by a data producer and a global identification code of the target business data; the data lake pushes the target business data and the global identification code of the target business data to a data channel in the data lake for storage; the data lake pushes the global identification code of the target business data to a verification channel in the data lake for storage, so that when a data consumer consumes data, the global identification code in the data channel is compared with the global identification code in the verification channel to check whether an abnormality occurs in the data in the data channel.

[0009] In some embodiments, the method also includes: the data lake receives a data consumption request sent by a data consumer, the data consumption request is used to request target consumption data from the data lake; the data lake responds to the data consumption request, obtains a global identification code corresponding to the target consumption data from the data channel and the verification channel; compares the global identification code of the target consumption data in the data channel with the global identification code of the target consumption data in the verification channel; if the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel, it is determined that an abnormality has occurred in the data in the data channel; if the global identification code of the target consumption data in the data channel is consistent with the global identification code of the target consumption data in the verification channel, it is determined that no abnormality has occurred in the data in the data channel, and the target consumption data is sent to the data consumer.

[0010] In some embodiments, the method further includes: the data lake pushes the target business data and the global identification code of the target business data to a file system for storage, so as to retransmit data to the data channel based on the data in the file system after determining that an abnormality occurs in the data in the data channel.

[0011] In some embodiments, the business data in the file system and the global identification code of the business data are synchronized to the target cache at a regular interval; wherein the method further includes: if it is determined that the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel, locating the global identification code of the target data where the abnormality occurs; the data lake extracts the target data from the target cache according to the global identification code of the target data; the data lake resends the target data to the data channel so that the data consumer obtains the complete target consumption data.

[0012] In some embodiments, the method further includes: if the target data is not in the target cache, notifying the data producer to supplement the target data.

[0013] In some embodiments, the method also includes: the data lake also pushes the target business data to the verification channel of the data lake for storage, so that when the data consumer consumes the data, the target business data in the data channel is compared with the target business data in the verification channel to verify whether the data in the data channel has been tampered with.

[0014] The embodiment of the present disclosure provides a data processing device, including: a business data acquisition module, a first push module and a second push module.

[0015] Among them, the business data acquisition module is used by the data lake to acquire the target business data produced by the data producer and the global identification code of the target business data; the first push module can be used by the data lake to push the target business data and the global identification code of the target business data to the data channel in the data lake for storage; the second push module can be used by the data lake to push the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to audit whether there is any abnormality in the data in the data channel.

[0016] An embodiment of the present disclosure provides an electronic device, which includes: a memory and a processor; the memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement any of the data processing methods described above.

[0017] The embodiment of the present disclosure provides a computer-readable storage medium on which computer program instructions are stored to implement any of the data processing methods described above.

[0018] The embodiment of the present disclosure provides a computer program product or a computer program, which includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the above data processing method.

[0019] The data processing method, device, electronic device, computer-readable storage medium and computer program product provided by the embodiments of the present disclosure can ensure the accuracy and consistency of data after entering the lake through a multi-channel data transmission mechanism and an automated audit and retransmission design.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0022] Figure 1 A schematic diagram of a scenario that can be applied to a data processing method or a data processing device in an embodiment of the present disclosure is shown.

[0023] Figure 2 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0024] Figure 3 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0025] Figure 4 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0026] Figure 5 A data supplementary transmission method is shown according to an exemplary embodiment.

[0027] Figure 6 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0028] Figure 7 It is a structural diagram of a data processing method according to an exemplary embodiment.

[0029] Figure 8 It is a block diagram of a data processing device according to an exemplary embodiment.

[0030] Fig. 9 A schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.

[0032] Those skilled in the art will appreciate that the embodiments of the present disclosure may be a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0033] The features, structures or characteristics described in the present disclosure may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0034] In the embodiments of the present disclosure, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0035] The accompanying drawings are only schematic diagrams of the present disclosure, and the same reference numerals in the drawings represent the same or similar parts, so their repeated description will be omitted. Some block diagrams shown in the accompanying drawings do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and steps, nor must they be executed in the order described. For example, some steps can be decomposed, and some steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0037] In the description of the present disclosure, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is only a kind of association relationship describing the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different; the terms "comprising", "including" and "having" are used to express the meaning of open inclusion and mean that in addition to the listed elements / components / etc., there may be other elements / components / etc.

[0038] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0039] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information and network security.

[0040] First, some terms involved in the embodiments of the present disclosure are explained below to facilitate understanding by those skilled in the art.

[0041] Data producers: that is, various business systems, which belong to the upstream data system. They can create new data or process the original data into new data or data products.

[0042] Data consumer: the data user, belonging to the data downstream system

[0043] Message: A message in a channel (topic) is the basic unit of data transfer, similar to a row of data or record in a database.

[0044] A data lake is a centralized storage area for storing, processing, and protecting large amounts of structured, semi-structured, and unstructured data. A data lake is a system or repository that stores data in a natural format. It can store all enterprise data of an enterprise in a unified manner, including raw data and transformed data for various tasks such as reporting, visualization, analysis, and machine learning.

[0045] The integration of data lake and warehouse is a new open architecture that fully combines the advantages of data lake and data warehouse. It is built on the low-cost data storage architecture of data lake, and inherits the data processing and data management functions of data warehouse, connecting the two systems of data lake and data warehouse, allowing data and computing to flow freely between lake and warehouse. Some people make a vivid metaphor of "integration of data lake and warehouse", just like building many small houses beside the lake (data lake), some of which can be responsible for data analysis, some for machine learning, some for audio and video retrieval, etc., and these data source streams can be easily obtained from the data lake.

[0046] The foregoing introduces some terms and concepts involved in the embodiments of the present disclosure. The following introduces the technical features involved in the embodiments of the present disclosure.

[0047] The purpose of this application is to provide a real-time data synchronization verification mechanism to reduce the inefficiency of manual auditing and solve the time-consuming and error-prone problems of traditional batch auditing. At the same time, by designing an automated data retransmission mechanism, the integrity and accuracy of data synchronization are ensured, creating a stable and universal real-time data synchronization engine.

[0048] The exemplary embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0049] Figure 1 A schematic diagram of a scenario that can be applied to a data processing method or a data processing device in an embodiment of the present disclosure is shown.

[0050] Please refer to Figure 1 , which shows a schematic diagram of an implementation environment provided by an exemplary embodiment of the present disclosure.

[0051] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0052] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, virtual reality devices, smart homes, etc.

[0053] The server 105 may be a server that provides various services, such as a background management server that provides support for devices operated by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received request data, and feed back the processing results to the terminal device.

[0054] The server can be an independent physical server or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as basic cloud computing services such as big data and artificial intelligence platforms, etc. The present disclosure does not impose any restrictions on this.

[0055] In some embodiments, the above-mentioned multiple servers can form a data lake; multiple servers and electronic devices can also jointly construct a data lake; multiple devices can also jointly construct a data lake. This application does not limit the construction entity of the data lake.

[0056] In some embodiments, the above-mentioned electronic device may be a data producer, and the above-mentioned server may also be a data producer, and this application does not impose any restrictions on this.

[0057] In some embodiments, the electronic device mentioned above may be a data consumer, and the server mentioned above may also be a data consumer, and this application does not impose any restrictions on this.

[0058] The server 105 constituting the data lake may, for example, obtain the target business data and the global identification code of the target business data produced by the data producer; the server 105 constituting the data lake may, for example, push the target business data and the global identification code of the target business data to the data channel in for storage; the server 105 constituting the data lake may, for example, push the global identification code of the target business data to the verification channel in for storage, so that when the data consumer consumes the data, the global identification code in the data channel can be compared with the global identification code in the verification channel to audit whether there is any abnormality in the data in the data channel.

[0059] It should be understood that Figure 1 The number of terminal devices, networks and servers in the figure is only for illustration. The server 105 may be a physical server or may be composed of multiple servers. According to actual needs, it may have any number of terminal devices, networks and servers.

[0060] Figure 2 is a flow chart of a data processing method according to an exemplary embodiment. The method provided by the embodiment of the present disclosure can be executed by any electronic device with computing and processing capabilities. For example, the method can be executed by the above Figure 1 The server or terminal device in the embodiment may be executed, or the server and the terminal device may be executed together. In the following embodiments, the server is used as the execution subject for example, but the present disclosure is not limited to this.

[0061] Reference Figure 2 The data processing method provided by the embodiment of the present disclosure may include the following steps.

[0062] Step S202: The data lake obtains the target business data produced by the data producer and the global identification code of the target business data.

[0063] In some embodiments, after producing business data or processing data products based on business data, the data producer will synchronize the data to the data lake so that the data lake can process and save it.

[0064] For example, within an enterprise, a business system may generate a lot of business data (such as shopping data in a shopping system). The business system will synchronize the business data it generates to the data lake for storage.

[0065] In some embodiments, after the data producer produces the target business data, it will send the target business data to the data lake, and the data lake will generate a global identification code for the target business data. The global identification code is used to uniquely identify the target business data in the data lake.

[0066] In some embodiments, after producing the target business data, the data producer generates a global identification code for the target business data.

[0067] In step S204, the data lake pushes the target business data and the global identification code of the target business data to the data channel in the data lake for storage.

[0068] In some embodiments, the data lake may include at least two channels: a data channel and a verification channel.

[0069] In some embodiments, the above data channel and verification channel can be different topics in Kafka. In Kafka, a topic can be a logical classification unit of a message, which carries messages and serves as a basic unit for message publishing and subscription. Each topic has a unique name to identify the type and purpose of the message.

[0070] Among them, the data channel can be used to store the business data synchronized by the data producer and the global identification code corresponding to the business data.

[0071] Among them, the verification channel is used to store the global identification code of the business data or store the business data and the global identification code, and this application does not impose any restrictions on this.

[0072] In some embodiments, whether in the verification channel or the data channel, the business data and / or the global identification code can be classified and stored according to business type, time, business content or production system, etc.

[0073] In some embodiments, the data in the data channel and the verification channel can be cleared at intervals; the data in the data channel and the verification channel can also be cleared within a certain time after the message is consumed. This application does not limit the timing of clearing data in the data channel and the verification channel.

[0074] Through the above method, the occupancy of consumption channels by data can be reduced and the utilization rate of consumption channels can be increased.

[0075] In step S206, the data lake pushes the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to check whether there is any abnormality in the data in the data channel.

[0076] In some embodiments, the data lake may push the global identification code of the target business data to a verification channel in the data lake for storage.

[0077] In some embodiments, the data lake may also push the target business data and the global identification code of the target business data to the verification channel for storage.

[0078] In some embodiments, if there is only a global identification code in the verification channel, then when the data consumer consumes data, the data lake will compare the global identification code in the data channel with the global identification code in the verification channel to audit whether there is any abnormality in the data in the data channel.

[0079] In some embodiments, if the verification channel contains both business data and a global identification code, then when the data consumer consumes data, not only will the global identification code in the data channel be compared with the global identification code in the verification channel to audit whether any abnormality has occurred in the data in the data channel, but also, if it is determined that an abnormality exists, the business data in the data channel will be compared with the business data in the verification channel to determine which data have an abnormality and what kind of abnormality has occurred.

[0080] The above method can ensure the accuracy and consistency of data after entering the lake through a multi-channel data transmission mechanism and automated auditing.

[0081] Figure 3 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0082] refer to Figure 3 , the above data processing method may include the following steps.

[0083] Step S302: The data lake receives a data consumption request sent by a data consumer, where the data consumption request is used to request target consumption data from the data lake.

[0084] In some embodiments, a data consumer may send a data consumption request to the data lake to request data.

[0085] In some embodiments, the target consumption data that the data consumer wants to consume can be determined according to the data consumption request. The target consumption data can be data produced by a production system in a certain period of time, or data produced by certain production systems. This application does not limit what the target consumption data is.

[0086] Step S304: The data lake responds to the data consumption request and obtains the global identification code corresponding to the target consumption data from the data channel and the verification channel.

[0087] In some embodiments, when the data lake receives the above data consumption request, it can determine the target consumption data that the data consumer wants and the global identification code corresponding to the target consumption data in the data channel according to the data consumption request. The target consumption data can be all or part of the data in the data channel, and this application does not limit this.

[0088] In some embodiments, after the data lake receives the above-mentioned data consumption request, it can determine the global identification code and / or target consumption data corresponding to the target consumption data that the data consumer wants to consume in the data channel according to the data consumption request.

[0089] Step S306, comparing the global identification code of the target consumption data in the data channel with the global identification code of the target consumption data in the verification channel.

[0090] In some embodiments, the data lake may compare the global identification code obtained from the data channel with the global identification code obtained from the verification channel to determine whether an abnormality (such as loss) occurs in the data channel for the target consumption data desired by the data consumer.

[0091] Step S308: If the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel, it is determined that an abnormality has occurred in the data in the data channel.

[0092] In some embodiments, if the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel, it is determined that an abnormality has occurred in the data in the data channel.

[0093] In some embodiments, after determining that an abnormality has occurred in the data in the data channel, the target consumption data obtained from the data channel can also be compared with the target consumption data obtained from the verification channel to determine which data has an abnormality, or determine what kind of abnormality has occurred in the data (such as tampering).

[0094] In some embodiments, when it is determined that data in a data channel is abnormal, the abnormal data is further located, and then the abnormal data is supplemented and an alarm is issued through the following method.

[0095] Step S310: If the global identification code of the target consumption data in the data channel is consistent with the global identification code of the target consumption data in the verification channel, it is determined that there is no abnormality in the data in the data channel, and the target consumption data is sent to the data consumer.

[0096] In some embodiments, if the global identification code of the target consumption data in the data channel is completely consistent with the global identification code of the target consumption data in the verification channel / the target consumption data in the data channel is completely consistent with the target consumption data in the verification channel, it is determined that there is no abnormality in the data in the data channel, and the target consumption data is sent to the data consumer in response to the consumption request.

[0097] Through the above method, the accuracy and consistency of data after entering the lake can be guaranteed through multi-channel data transmission mechanism and automated auditing.

[0098] Figure 4 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0099] refer to Figure 4 , the above data processing method may include the following steps.

[0100] Step S402: The data lake obtains the target business data produced by the data producer and the global identification code of the target business data.

[0101] Step S404: The data lake pushes the target business data and the global identification code of the target business data to the data channel in the data lake for storage.

[0102] In step S406, the data lake pushes the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to check whether there is any abnormality in the data in the data channel.

[0103] In step S408, the data lake pushes the target business data and the global identification code of the target business data to the file system for storage, so as to retransmit the data to the data channel according to the data in the file system after determining that an abnormality occurs in the data in the data channel.

[0104] In some embodiments, the data lake may push the target business data and the global identification code of the target business data to a file system for storage.

[0105] The file system can be used to store data. The file system can periodically push data to the cache of the data lake so that after determining that the data in the data channel is abnormal, the cache automatically retransmits the data to the data channel.

[0106] Figure 5 A data supplementary transmission method is shown according to an exemplary embodiment.

[0107] In some embodiments, the business data and the global identification code of the business data in the file system may be synchronized to the target cache at a regular interval.

[0108] refer to Figure 5 The above data supplement transmission method may include the following steps.

[0109] Step S502: if it is determined that the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel, the global identification code of the target data where the abnormality occurs is located.

[0110] Step S504: The data lake extracts the target data from the target cache according to the global identification code of the target data.

[0111] Step S506: The data lake resends the target data to the data channel so that the data consumer can obtain the complete target consumption data.

[0112] Step S508: If the target data does not exist in the target cache, the data producer is notified to supplement the target data.

[0113] Through the above method, target data with abnormalities in the data channel can be retransmitted in a timely manner.

[0114] Figure 6 The invention is a flowchart of a data processing method according to an exemplary embodiment.

[0115] refer to Figure 6 , the above data processing method may include the following steps.

[0116] Step S602: The data lake obtains the target business data produced by the data producer and the global identification code of the target business data.

[0117] Step S604: The data lake pushes the target business data and the global identification code of the target business data to the data channel in the data lake for storage.

[0118] In step S606, the data lake pushes the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to check whether there is any abnormality in the data in the data channel.

[0119] In step S608, the data lake also pushes the target business data to the verification channel of the data lake for storage, so that when the data consumer consumes the data, the target business data in the data channel is compared with the target business data in the verification channel to verify whether the data in the data channel has been tampered with.

[0120] Through the above method, it is possible to timely and accurately determine whether the data in the data channel is abnormal and accurately determine which data is abnormal, so as to perform data retransmission to achieve accuracy and timeliness of data synchronization.

[0121] Below, this application will explain and illustrate the data processing method in conjunction with specific embodiments.

[0122] Figure 7 It is a structural diagram of a data processing method according to an exemplary embodiment.

[0123] refer to Figure 7 The data processing structure corresponding to the above data processing method may include a data producer (upstream system) 701, a data lake 702, and a consumer (downstream system) 703.

[0124] refer to Figure 7 The data producer 701 can produce business data and send the business data to the data lake for storage.

[0125] refer to Figure 7 The above data processing method proposes a mechanism for dual-channel data transmission in the data lake: in the data lake 702, each piece of data is sent to two different Kafka channels: the data channel (data topic Kafka) and the verification channel (account reconciliation topic Kafka). The data channel stores the original message and the unique identification code, and the verification channel stores the unique identification and the number of message records. The unique identification codes on both sides correspond to each other. The data channel of each topic represents a logical business data set, and each message carries a globally unique identification code for subsequent comparison and retrieval.

[0126] refer to Figure 7 ,The above data processing method proposes a real-time audit and anomaly detection mechanism: the data consumer 703 consumes messages from the data channel and the verification channel at the same time, and performs a real-time comparison based on the globally unique identification code. If the data of the two channels differ or have a delay within a specific time window, the system will generate an audit log and start the anomaly handling process.

[0127] refer to Figure 7The above data processing method also proposes an offline caching and automatic retransmission mechanism: when synchronizing data, the data lake will also write a copy of each data to HBase for caching. When it is detected that the data synchronization is incomplete or abnormal, the corresponding copy data is automatically pulled from HBase and resent to ensure the integrity of data transmission. Or when synchronizing data, the data lake will also write a copy of the data to the HDFS file system. The HDFS file system will periodically update the data to HBase. When it is detected that the data synchronization is incomplete or abnormal, the corresponding copy data is automatically pulled from HBase and resent to ensure the integrity of data transmission.

[0128] refer to Figure 7 The above data processing method also proposes an exception handling and notification mechanism: when missing data cannot be found in HBase, the system will notify the source system to re-enter the data through the exception message mechanism. The source system resends the missing data according to the notification to ensure that the data can finally enter the data lake correctly.

[0129] Below, this application will combine Figure 7 The above data processing method is described in detail.

[0130] 1. Upstream system.

[0131] 1) Data generation and transmission.

[0132] After the upstream system generates business data, a unique global identification code is added to each message in the Topic. Use asynchronous IO (input / output) to push the message with the unique identification code to the data channel Kafka (asynchronous IO is suitable for scenarios that need to process a large number of concurrent IO requests and is more efficient). At the same time, in order to improve data reliability, the identification code is pushed to the verification channel Kafka through synchronous IO. This synchronous IO operation is to ensure that the sending timing of the verification data and the data channel are strictly synchronized. The original copy of the message is saved to the HDFS file system and regularly synchronized to HBase for subsequent abnormal retransmission.

[0133] 2. Data lake.

[0134] 1) Message channel design.

[0135] The data channel Kafka stores complete business data, and the verification channel Kafka only stores message identification codes for reconciliation.

[0136] Regularly clean up expired data in both to avoid data backlog. The cleanup mechanism is based on the life cycle of the message, for example, clearing the message within a certain period of time after it is consumed to prevent resource waste.

[0137] 2) Real-time audit service.

[0138] The data consumer consumes messages from the data channel Kafka and the verification channel Kafka concurrently, and matches and audits them through the globally unique identification code. If there is an inconsistency in the identification code within the set time window, the system will trigger an alarm and record an audit log. The audit mechanism not only detects data loss, but also discovers potential data tampering problems by comparing the message content.

[0139] 3) Automatic retransmission mechanism.

[0140] When an audit anomaly is found, the system first extracts a copy of the data from HBase and resends it to the data channel Kafka to ensure that the data can be consumed completely. If the corresponding data cannot be found in HBase, the system will notify the source system through an exception message and require the source system to re-enter the data.

[0141] 4) Data cleaning and maintenance.

[0142] The data in the data channel and the verification channel will be automatically cleaned up within the set time period to prevent the excessive amount of data in the message queue from affecting performance. The cleanup strategy is based on the retention time after data consumption is completed.

[0143] 5) Real-time delay monitoring and data visualization.

[0144] The system provides real-time latency monitoring, and timely grasps the data synchronization latency by analyzing the transmission time of data from the source system to the data lake. It also provides visual audit results, including data fluctuations, accuracy, and latency trends for each time period.

[0145] The above embodiments have at least the following technical points.

[0146] 1. Dual-channel data transmission design: The data channel and the verification channel are used to separate the transmission of messages and their identification codes, thus achieving real-time auditing and accuracy assurance. The data stored in the verification channel is only the global identification code, which greatly reduces storage redundancy and storage costs.

[0147] 2. Globally unique message ID: Each message carries a unique global identification code to ensure fast and accurate matching and verification in multi-channel synchronization.

[0148] 3. Offline caching and automatic retransmission mechanism: HBase caches data for a certain period of time to ensure automatic retransmission when data is lost or transmitted abnormally, thereby improving the stability of data synchronization.

[0149] 4. Real-time audit and alarm: The system has real-time audit and abnormal alarm capabilities. When data synchronization is incomplete or there is a delay, it can generate audit logs in time and trigger automatic remediation mechanisms.

[0150] The above embodiments have at least the following beneficial effects.

[0151] 1. Real-time audit and delay monitoring: Through real-time verification, the system can quickly detect delays and losses in data synchronization, and ensure the accuracy of each synchronization through audit log records.

[0152] 2. Automatic retransmission to ensure consistency: Through the retransmission mechanism of HBase cache, even if an exception occurs during data transmission, it can be automatically retransmitted, reducing the risk of data loss.

[0153] 3. Data synchronization visualization and insight: Through the visual display of audit, users can grasp the status and accuracy of data synchronization in real time and improve the efficiency of system operation and maintenance.

[0154] It should be particularly pointed out that the various steps in the various embodiments of the above data processing method can be mutually crossed, replaced, added, or deleted. Therefore, these reasonable permutations, combinations, and transformations of the data processing method should also fall within the scope of protection of the present disclosure, and the scope of protection of the present disclosure should not be limited to the embodiments.

[0155] Based on the same inventive concept, the present disclosure also provides a data processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0156] Figure 8 is a block diagram of a data processing device according to an exemplary embodiment. Figure 8 The data processing device 800 provided in the embodiment of the present disclosure may include: a business data acquisition module 801, a first push module 802 and a second push module 803.

[0157] Among them, the business data acquisition module 801 can be used in the data lake to acquire the target business data produced by the data producer and the global identification code of the target business data; the first push module 802 can be used in the data lake to push the target business data and the global identification code of the target business data to the data channel in the data lake for storage; the second push module 803 can be used in the data lake to push the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to audit whether there is any abnormality in the data in the data channel.

[0158] It should be noted that the business data acquisition module 801, the first push module 802 and the second push module 803 correspond to S202 to S206 in the method embodiment, and the examples and application scenarios implemented by the modules and the corresponding steps are the same, but are not limited to the contents disclosed in the method embodiment. It should be noted that the modules as part of the device can be executed in a computer system such as a set of computer executable instructions.

[0159] In some embodiments, the data processing device 800 may further include: a consumption request acquisition module, a first identification code acquisition module, a second identification code acquisition module, a first comparison module and a second comparison module.

[0160] Among them, the consumption request acquisition module is used for the data lake to receive the data consumption request sent by the data consumer, and the data consumption request is used to request the target consumption data from the data lake; the first identification code acquisition module is used for the data lake to respond to the data consumption request and obtain the global identification code corresponding to the target consumption data from the data channel and the verification channel; the second identification code acquisition module is used to compare the global identification code of the target consumption data in the data channel with the global identification code of the target consumption data in the verification channel; the first comparison module is used to determine that an abnormality has occurred in the data in the data channel if the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel; the second comparison module is used to determine that no abnormality has occurred in the data in the data channel if the global identification code of the target consumption data in the data channel is consistent with the global identification code of the target consumption data in the verification channel, and send the target consumption data to the data consumer.

[0161] In some embodiments, the data processing device 800 may further include: a third push module.

[0162] Among them, the third push module can be used by the data lake to push the target business data and the global identification code of the target business data to the file system for storage, so as to supplement the data to the data channel according to the data in the file system after determining that an abnormality occurs in the data in the data channel.

[0163] In some embodiments, the business data in the file system and the global identification code of the business data are synchronized to the target cache at a regular interval; wherein the data processing device 800 may further include: an abnormal data locating module, a cache extracting module and a resending module.

[0164] Among them, the abnormal data locating module is used to locate the global identification code of the target data where the abnormality occurs if it is determined that the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel; the cache extraction module is used by the data lake to extract the target data from the target cache according to the global identification code of the target data; the resending module is used by the data lake to resend the target data to the data channel so that the data consumer obtains complete target consumption data.

[0165] In some embodiments, the data processing device 800 may further include: a supplementary recording notification module.

[0166] The supplementary recording notification module is used to notify the data producer to supplement the target data if the target data is not in the target cache.

[0167] In some embodiments, the data processing device 800 may further include: a business data pushing module.

[0168] Among them, the business data push module can be used in the data lake to push the target business data to the verification channel of the data lake for storage, so that when the data consumer consumes the data, the target business data in the data channel is compared with the target business data in the verification channel to verify whether the data in the data channel has been tampered with.

[0169] Since the functions of the device 800 have been described in detail in the corresponding method embodiments, the present disclosure will not elaborate on them here.

[0170] The modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The modules described may also be set in a processor. The names of these modules do not, in some cases, constitute limitations on the modules themselves.

[0171] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a part of a module or program segment, and a part of the above-mentioned module or program segment contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer program instructions.

[0172] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0173] Fig. 9 FIG. 1 shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present disclosure. It should be noted that: Fig. 9 The electronic device 900 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0174] like Fig. 9 As shown, the electronic device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0175] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0176] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains computer program instructions for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 909, and / or installed from a removable medium 911. When the computer program is executed by a central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present disclosure are executed.

[0177] It should be noted that the computer-readable storage medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable computer program instructions. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The computer program instructions contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0178] As another aspect, the present disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by a device, the device can realize functions including: the data lake obtains the target business data and the global identification code of the target business data produced by the data producer; the data lake pushes the target business data and the global identification code of the target business data to the data channel in the data lake for storage; the data lake pushes the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to audit whether the data in the data channel is abnormal.

[0179] According to one aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer program instructions, the computer program instructions being stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0180] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of computer program instructions for enabling an electronic device (which can be a server or a terminal device, etc.) to execute the method according to the embodiment of the present disclosure.

[0181] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0182] It should be understood that the present disclosure is not limited to the detailed structures, drawings or implementations shown herein, but rather the present disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data processing method, characterized in that: include: The data lake obtains the target business data produced by the data producer and the global identification code of the target business data; The data lake pushes the target business data and the global identification code of the target business data to a data channel in the data lake for storage; The data lake pushes the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to audit whether there is any abnormality in the data in the data channel.

2. The method according to claim 1, characterized in that: The method further comprises: The data lake receives a data consumption request sent by a data consumer, where the data consumption request is used to request target consumption data from the data lake; The data lake obtains, in response to the data consumption request, a global identification code corresponding to the target consumption data from the data channel and the verification channel; Comparing the global identification code of the target consumption data in the data channel with the global identification code of the target consumption data in the verification channel; If the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel, it is determined that an abnormality has occurred in the data in the data channel; If the global identification code of the target consumption data in the data channel is consistent with the global identification code of the target consumption data in the verification channel, it is determined that no abnormality has occurred in the data in the data channel, and the target consumption data is sent to the data consumer.

3. The method according to claim 1, characterized in that: The method further comprises: The data lake pushes the target business data and the global identification code of the target business data to the file system for storage, so as to retransmit the data to the data channel according to the data in the file system after determining that an abnormality occurs in the data in the data channel.

4. The method according to claim 3, characterized in that: Synchronizing the business data in the file system and the global identification code of the business data to the target cache at a regular time; wherein the method further comprises: If it is determined that the global identification code of the target consumption data in the data channel is inconsistent with the global identification code of the target consumption data in the verification channel, locating the global identification code of the target data where the abnormality occurs; The data lake extracts the target data from the target cache according to the global identification code of the target data; The data lake resends the target data to the data channel so that the data consumer obtains complete target consumption data.

5. The method according to claim 4, characterized in that: The method further comprises: If the target data is not in the target cache, the data producer is notified to supplement the target data.

6. The method according to claim 1, characterized in that: The method further comprises: The data lake also pushes the target business data to the verification channel of the data lake for storage, so that when the data consumer consumes the data, the target business data in the data channel is compared with the target business data in the verification channel to verify whether the data in the data channel has been tampered with.

7. A data processing device, characterized in that: include: A business data acquisition module is used for the data lake to acquire the target business data produced by the data producer and the global identification code of the target business data; A first push module, configured for the data lake to push the target business data and the global identification code of the target business data to a data channel in the data lake for storage; The second push module is used for the data lake to push the global identification code of the target business data to the verification channel in the data lake for storage, so that when the data consumer consumes the data, the global identification code in the data channel is compared with the global identification code in the verification channel to audit whether there is any abnormality in the data in the data channel.

8. An electronic device, characterized in that: include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the data processing method according to any one of claims 1 to 7.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the data processing method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising computer program instructions, wherein the computer program instructions are stored in a computer-readable storage medium, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images