Data processing method and device, equipment, computer medium and product

Abstract

The invention discloses a data processing method, device and equipment, a computer medium and a product, and is applied to the field of data processing.The method comprises the steps that in response to detected business operation of data change of an upstream business system, business data are collected from a database log; dividing the business data into real-time business data and dimension table business data based on the updating frequency of the business data; sending the real-time business data and the dimension table business data to a streaming computing platform through a database acquisition component; storing the dimension table service data in a preset initial increment dimension table cache through the streaming computing platform to obtain an updated increment dimension table cache; on the basis of the service identification information, associating the real-time service data with dimension table service data in the update increment dimension table cache to obtain service result data; and the streaming computing platform sends the service result data to a data subscription demander through the message middleware. According to the method, data processing efficiency is improved, and computing resources are saved.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to a data processing method, apparatus, device, computer storage medium, and computer program product. Background Technology

[0002] With the development of technology, more and more businesses are placing higher demands on the timeliness of data. How to improve data processing efficiency and save resources while maintaining timeliness has become an increasingly important topic.

[0003] Most existing business data processing solutions collect business data in real time, and then perform statistics and analysis on the collected real-time business data. This method of real-time collection and real-time processing not only causes significant resource consumption, but also reduces data processing efficiency.

[0004] Therefore, how to provide a data processing method that improves data processing efficiency while saving computing resources is a technical problem that continues to be solved. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, computer storage medium, and computer program product that can improve data processing efficiency while saving computing resources.

[0006] In a first aspect, embodiments of this application provide a data processing method applied to a data processing system, the data processing system comprising: a database acquisition component, a streaming computing platform, a message middleware connected to the streaming computing platform via message communication, and a data subscription demand party; the method comprising: In response to a business operation that detects a data change in the upstream business system, business data is collected from the database logs through the database acquisition component; based on the update frequency of the business data, the business data is divided into real-time business data and dimension table business data; the real-time business data is data whose update frequency is greater than a preset threshold, and the dimension table business data is data whose update frequency is not greater than a preset threshold; the real-time business data and the dimension table business data are sent to the streaming computing platform through the database acquisition component; The dimension table business data is stored in a preset initial incremental dimension table cache through the streaming computing platform to obtain an updated incremental dimension table cache. Based on the business identification information, the real-time business data is associated with the dimension table business data in the updated incremental dimension table cache through the streaming computing platform to obtain business result data; The streaming computing platform sends the business result data to the data subscription demander through the message middleware.

[0007] In some possible implementations, before sending the real-time business data and the dimension table business data to the streaming computing platform, the method further includes: The data in the real-time business data and the dimension table business data are subjected to quality verification and risk control rule matching; If the data quality checks of the real-time business data and the dimension table business data pass and no risk control rules are triggered, the real-time business data and the dimension table business data are sent to the streaming computing platform.

[0008] In some possible implementations, the method further includes: If the data quality check of the real-time business data and the dimension table business data fails, the real-time business data and the dimension table business data will be routed to a preset error isolation area. Record error details of data routed to the preset error isolation zone and output data quality alarm information.

[0009] In some possible implementations, after outputting the data quality alarm information, the method further includes: Based on the error details, a repair task is created for the data routed to the preset error isolation zone; In response to receiving a data repair instruction for the repair task, the data routed to the preset error isolation area is repaired to obtain the repaired data; Based on the repaired data, the quality verification and risk control rule matching of the data in the real-time business data and the dimension table business data are performed.

[0010] In some possible implementations, the method further includes: If the risk control rule is triggered by data in the real-time business data and the dimension table business data, the real-time business data and the dimension table business data will be routed to a preset risk control event area. Based on the risk event level of the data routed to the preset risk control event zone, a risk control alarm corresponding to the risk event level is generated.

[0011] In some possible implementations, the streaming computing platform sends the business result data to the data subscription requester through the message middleware, including: Obtain the topic of the message middleware; Based on the association between the topic of the business result data and the topic of the message middleware, select the message middleware corresponding to the topic of the business result data; The business result data is published to the corresponding message middleware so that the data subscription demander can obtain the corresponding business result data according to the topic.

[0012] In some possible implementations, the streaming computing platform sends the business result data to the data subscription requester through the message middleware, including: Detect the data traffic corresponding to the business result data; If the data flow exceeds a preset threshold, the business result data is stored in the preset data encapsulation interface of the streaming computing platform, and a reminder signal is sent to the data subscription demander to remind the data subscription demander to improve data consumption efficiency.

[0013] Secondly, embodiments of this application provide a data processing apparatus applied to a data processing system, the data processing system comprising: a database acquisition component, a streaming computing platform, a message middleware connected to the streaming computing platform via message communication, and a data subscription demand party; the apparatus comprising: The acquisition module is used to acquire business data from the database logs through the database acquisition component in response to a business operation that detects a data change in the upstream business system; the segmentation module is used to segment the business data into real-time business data and dimension table business data based on the update frequency of the business data; the real-time business data is data whose update frequency is greater than a preset threshold, and the dimension table business data is data whose update frequency is not greater than a preset threshold; the sending module is used to send the real-time business data and the dimension table business data to the streaming computing platform through the database acquisition component; The storage module is used to store the dimension table business data in a preset initial incremental dimension table cache through the streaming computing platform to obtain an updated incremental dimension table cache. The association module is used to associate the real-time business data with the dimension table business data in the updated incremental dimension table cache based on the business identification information through the streaming computing platform to obtain business result data; The sending module is also used by the streaming computing platform to send the business result data to the data subscription demander through the message middleware.

[0014] Thirdly, embodiments of this application provide a data processing apparatus, the apparatus comprising: A processor and a memory storing computer program instructions; a data processing method that implements any of the above when the processor executes the computer program instructions.

[0015] Fourthly, embodiments of this application provide a computer storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the data processing method described above is implemented.

[0016] Fifthly, embodiments of this application provide a computer program product in which instructions, when executed by a processor of an electronic device, enable the electronic device to perform any of the above-mentioned data processing methods.

[0017] The data processing method, apparatus, device, computer storage medium, and computer program product of this application embodiment can divide business data into high-speed changing real-time business data and low-speed changing dimension table business data based on the update frequency of the business data. This avoids the problems of high resource consumption and low data processing efficiency caused by real-time acquisition and processing of business data in traditional solutions. Furthermore, by constructing a dedicated incremental dimension table cache and storing dimension table data in the incremental dimension table cache, the high-latency external database query that originally needed to be performed during processing is transformed into a millisecond-level memory query performed within the streaming computing platform. This greatly reduces the load pressure on the business database and improves the correlation efficiency between real-time business data and dimension table business data, achieving second-level or even millisecond-level output of business result data, thus greatly improving data processing efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this application; Figure 2 This is a schematic flowchart of a data processing method provided in one embodiment of this application; Figure 3 This is a schematic diagram of the structure of a data processing apparatus provided in another embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended only to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0023] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0024] Most existing business data processing solutions collect business data in real time, and then perform statistics and analysis on the collected real-time business data. This method of real-time collection and real-time processing not only causes significant resource consumption, but also reduces data processing efficiency.

[0025] To address the problems of the prior art, embodiments of the present invention provide a data processing method, apparatus, device, and computer storage medium.

[0026] The data processing method provided in the embodiments of the present invention will be described below first.

[0027] This data processing method is applied to data processing systems, such as... Figure 1As shown, the data processing system 100 includes: a database acquisition component 110, a streaming computing platform 120, a message middleware 130 connected to the streaming computing platform 120 for message communication, and a data subscription demander 140.

[0028] Database acquisition component 110 is responsible for real-time monitoring and capturing every change in the upstream business system database (such as data insertion INSERT, update UPDATE, and deletion DELETE).

[0029] The streaming computing platform 120 receives raw data streams from the database acquisition component 110, cleans and classifies the data, filters invalid data, and divides it into real-time business data and dimension table business data according to update frequency. It builds and updates the dimension table cache in memory in real time, and associates the real-time business data with the dimension table cache to assemble the final business result data. Simultaneously, the streaming computing platform 120 can also perform tasks such as risk control rule matching.

[0030] Message middleware 130 isolates data producers (streaming computing platforms) and consumers (data subscription demanders), ensuring that failures or scaling on one side do not directly affect the other. Message middleware 130 acts as a data buffer, balancing the difference between production and consumption speeds to handle traffic peaks, while providing a data persistence mechanism to ensure data is not lost.

[0031] Data subscription demanders 140 consume and use processed, high-quality, real-time results data.

[0032] Figure 2 A schematic flowchart of a data processing method according to an embodiment of the present invention is shown. Figure 2 As shown, the method may include the following steps: S210. In response to a business operation that detects a data change in the upstream business system, business data is collected from the database logs through the database collection component.

[0033] For example, the database acquisition component 110 is responsible for monitoring and capturing every change (such as INSERT, UPDATE, DELETE) in the upstream business system database in real time. For instance, a user completes a payment, creates a new order, or an administrator modifies a product price. By reading the database's transaction log to obtain data changes, low-latency and low-impact data acquisition is achieved.

[0034] S220. Based on the update frequency of business data, business data is divided into real-time business data and dimension table business data.

[0035] For example, real-time business data refers to data whose update frequency is greater than a preset threshold, while dimension table business data refers to data whose update frequency is not greater than a preset threshold.

[0036] Real-time business data can include transaction logs, user click logs, server monitoring metrics, etc., and is characterized by high throughput and low latency. Dimension table business data can include user information tables (username, level), product information tables (product name, category), merchant information tables, etc., which are relatively stable but need to be accurate, and therefore require persistence and consistency.

[0037] S230: Send real-time business data and dimension table business data to the streaming computing platform through the database acquisition component.

[0038] S240. The dimension table business data is stored in the preset initial incremental dimension table cache through the streaming computing platform to obtain the updated incremental dimension table cache.

[0039] An incremental dimension table cache is a copy of dimension table data stored in the Flink streaming engine's memory or in external high-speed storage (such as Redis). The data processing system 100 pre-builds an initial incremental dimension table cache containing all dimension table data. The streaming platform 120 updates only those dimension table records that have changed using Change Data Capture (CDC) technology.

[0040] Change Data Capture (CDC) is a technique used to identify and track changes (inserts, deletes, and updates) that occur in a database. It typically achieves this by reading the database's transaction log, capturing changes in near real-time rather than by polling tables.

[0041] S250. Based on the business identification information, the real-time business data is associated with the dimension table business data in the incremental dimension table cache through the streaming computing platform to obtain the business result data.

[0042] After receiving real-time business data, the streaming computing platform 120 extracts its business identification information (such as user_id:XXX), and then directly queries the corresponding detailed information in the local incremental dimension table cache. It assembles the information from both sides into a complete and directly usable data, namely business result data.

[0043] The S260 streaming computing platform sends business result data to data subscription demanders through message middleware.

[0044] It's important to note that the streaming computing platform 120 is only responsible for producing data and doesn't need to worry about who consumes it or how. Downstream demanders only need to obtain data from the message middleware 130. The message middleware 130 (such as Kafka) acts as a buffer; when downstream consumption capacity is insufficient, data will accumulate in the queue, preventing the computing platform from being overwhelmed. Simultaneously, it ensures that no data is lost.

[0045] This method collects business data in real time through a database acquisition component, divides the data into real-time business data and dimension table business data based on the update frequency, and stores the dimension table data in an incremental dimension table cache. Finally, it associates the real-time data with the cached dimension table data, obtains the results, and distributes them to the requesting party through a message middleware. By dividing the business data into high-speed changing real-time business data and low-speed changing dimension table business data, it avoids the high resource overhead and low data processing efficiency caused by real-time acquisition and processing of business data in traditional solutions. Furthermore, by building a dedicated incremental dimension table cache and storing the dimension table data there, it transforms the high-latency external database queries that originally needed to be performed during processing into millisecond-level in-memory queries performed within the streaming computing platform. This greatly reduces the load on the business database and improves the efficiency of associating real-time business data with dimension table business data, achieving second-level or even millisecond-level output of business result data, thus significantly improving data processing efficiency.

[0046] In some embodiments, before sending real-time business data and dimension table business data to the streaming computing platform, the method may further include: Perform quality verification and risk control rule matching on data in real-time business data and dimension table business data; If the data quality of real-time business data and dimension table business data passes the inspection and no risk control rules are triggered, the real-time business data and dimension table business data will be sent to the streaming computing platform.

[0047] It should be noted that before sending real-time business data and dimension table business data to the streaming computing platform, a preliminary step of data quality verification and risk control rule matching was added. This ensures that only data that passes the verification and does not trigger risk control will enter the core streaming computing platform, effectively preventing dirty and risky data from polluting and impacting subsequent complex calculations, and improving the quality of the final result data and the security of the system.

[0048] In some embodiments, if the data quality inspection of real-time business data and dimension table business data fails, the real-time business data and dimension table business data can be routed to a preset error isolation area; error details of the data routed to the preset error isolation area are recorded, and data quality alarm information is output.

[0049] If data quality checks fail in real-time business data and dimension table business data, the data processing system can automatically redirect non-compliant data (e.g., missing fields, incorrect formatting, out-of-bounds values) to a dedicated, independent storage area—the error isolation zone. This area is physically or logically completely separated from the "normal pipeline" that processes normal data, preventing a few "bad data" entries from contaminating the entire data processing pipeline and avoiding errors, stalls, or crashes in core stream processing jobs. This ensures the smooth and stable operation of the main business data channel.

[0050] For example, the data processing system generates a detailed "diagnostic report" for each isolated erroneous data entry. This report records the data's content, source, the specific rules that triggered the error (e.g., "user ID field is empty"), and metadata such as the time of occurrence. It also outputs data quality alarm information. This alarm can be promptly notified to data operations or development personnel via email, SMS, monitoring dashboards, or integration into the operations and maintenance management system, enabling them to quickly locate the problem.

[0051] After outputting data quality alarm information, as an example, a repair task can be created for data routed to a preset error isolation zone based on the error details; in response to receiving a data repair instruction for the repair task, the data routed to the preset error isolation zone is repaired to obtain the repaired data; based on the repaired data, quality verification and risk control rule matching are performed on the data in real-time business data and dimension table business data.

[0052] For example, the data processing system can also create repair tasks for data in the error isolation zone to fix the erroneous data and put the repaired data back into the starting point of the data processing flow, i.e., to re-perform quality verification and risk control rule matching. By upgrading simple error isolation to a "data governance closed loop," it ensures that data is not permanently lost due to temporary errors, maximizes the recovery of data caused by quality problems, significantly improves data integrity and utilization, and realizes the self-repair capability of the data system.

[0053] As an example, when risk control rules are triggered by data in real-time business data and dimension table business data, the real-time business data and dimension table business data can be routed to a preset risk control event area; based on the risk event level of the data routed to the preset risk control event area, a risk control alert corresponding to the risk event level is generated.

[0054] When the data processing system detects a pattern in the data stream that matches preset risk control rules (e.g., multiple large transfers from the same account within one second, an abnormal surge in transaction volume for a merchant, or a transaction location that does not match the user's usual location), it immediately separates that data item or batch from the main data processing pipeline and directs it to a dedicated "risk control event area." This ensures that high-risk events enter a special emergency channel and are not mixed in with normal business data, avoiding delays. Simultaneously, it guarantees that complex analysis of risk control events and normal business processing can proceed concurrently without blocking each other, ensuring that the performance of core business operations is not affected by risk control scanning.

[0055] For example, the data processing system also analyzes data entering the risk control event zone, assesses its risk severity, and dynamically generates alerts of different levels. For high-risk events, an "emergency alert" may be generated, triggering automatic actions (such as temporary account freezing), and risk control personnel will be immediately notified via phone and SMS. For medium-risk events, a "warning" may be generated and sent to the risk control work order system, requiring manual review. For low-risk events, only a "prompt message" may be generated and logged for subsequent analysis. This precise response avoids a "one-size-fits-all" approach to alerts, allowing risk control resources to be concentrated on the most dangerous events, achieving optimized resource allocation and precise emergency response. Different alert levels correspond to different handling procedures, standardizing and automating response actions, significantly improving the efficiency of risk response.

[0056] In some embodiments, S260 may include: Get the topic of the message middleware; Based on the relationship between the topic of the business result data and the topic of the message middleware, select the message middleware that corresponds to the topic of the business result data; The business results data is published to the corresponding message middleware so that data subscribers can obtain the corresponding business results data according to the topic.

[0057] For example, the message middleware pre-defines multiple topics, such as those for real-time monitoring dashboards, risk control analysis, and data warehouse ingestion. These topics determine the available output channels. The data processing system, based on pre-configured mapping rules, determines which topic(s) to send the currently processed "business result data" to, and pushes the business result data to the corresponding topic's message middleware.

[0058] Different result data are published to different channels of the message middleware based on the topic. Downstream demanders can subscribe to specific topics according to their own needs, which decouples data distribution and consumption and greatly improves the flexibility and scalability of the system architecture.

[0059] In some embodiments, S260 may further include: Data traffic corresponding to the detection business results data; When the data traffic exceeds a preset threshold, the business results data is stored in the preset data encapsulation interface of the streaming computing platform, and a reminder signal is sent to the data subscription demander to remind the data subscription demander to improve data consumption efficiency.

[0060] The data processing system continuously monitors the data flow rate from the streaming computing platform to the message middleware, typically referring to the production throughput. Simultaneously, the data processing system indirectly senses or directly obtains the downstream consumption throughput from the message middleware. When the data processing system determines that the production rate consistently exceeds the consumption rate, causing data backlog, it will not indiscriminately push data into the message middleware (this could lead to memory overflow). Instead, the data processing system will activate an emergency buffer mechanism—temporarily storing the business result data to be distributed in a pre-defined data encapsulation interface within the streaming computing platform. This data encapsulation interface is a controlled, limited-size temporary storage area within the streaming computing platform (e.g., Flink's managed memory or an internal blocking queue). This data encapsulation interface protects the message middleware, preventing downstream consumption bottlenecks from propagating upwards and ultimately overwhelming the message middleware, causing the entire data channel to crash. Simultaneously with activating the internal buffer, the data processing system proactively sends a reminder signal to data subscribers with slow consumption, prompting them to improve data consumption efficiency, thus dynamically driving downstream systems to scale up or optimize.

[0061] An adaptive load buffering mechanism is provided to cope with data traffic surges. When traffic is too high, it temporarily stores data and alerts downstream devices to accelerate consumption. This ensures that the main processing links are not overwhelmed and no data is lost in the event of a sudden surge in data volume, thus guaranteeing system continuity and high availability.

[0062] The data processing method, apparatus, device, computer storage medium, and computer program product of this application embodiment can divide business data into high-speed changing real-time business data and low-speed changing dimension table business data based on the update frequency of the business data. This avoids the problems of high resource consumption and low data processing efficiency caused by real-time acquisition and processing of business data in traditional solutions. Furthermore, by constructing a dedicated incremental dimension table cache and storing dimension table data in the incremental dimension table cache, the high-latency external database query that originally needed to be performed during processing is transformed into a millisecond-level memory query performed within the streaming computing platform. This greatly reduces the load pressure on the business database and improves the correlation efficiency between real-time business data and dimension table business data, achieving second-level or even millisecond-level output of business result data, thus greatly improving data processing efficiency.

[0063] Based on the data processing method provided in the above embodiments, this application also provides specific implementations of a data processing apparatus. Please refer to the following embodiments.

[0064] First see Figure 3 The data processing device is applied to a data processing system, which includes: a database acquisition component, a streaming computing platform, a message middleware connected to the streaming computing platform for message communication, and a data subscription demander; the data processing device 300 provided in this embodiment includes: The acquisition module 310 is used to collect business data from the database logs through the database acquisition component in response to the detection of data changes in the upstream business system. The segmentation module 320 is used to divide the business data into real-time business data and dimension table business data based on the update frequency of the business data. Real-time business data is data whose update frequency is greater than a preset threshold, and dimension table business data is data whose update frequency is not greater than a preset threshold. The sending module 330 is used to send the real-time business data and dimension table business data to the streaming computing platform through the database acquisition component. Storage module 340 is used to store dimension table business data in a preset initial incremental dimension table cache through a streaming computing platform to obtain an updated incremental dimension table cache. The association module 350 is used to associate real-time business data with dimension table business data in the incremental dimension table cache through the streaming computing platform based on business identification information to obtain business result data; The sending module 330 is also used by the streaming computing platform to send business result data to data subscription demanders through message middleware.

[0065] In some possible implementations, before sending real-time business data and dimension table business data to the streaming computing platform, the sending module 330 is also used for: Perform quality verification and risk control rule matching on data in real-time business data and dimension table business data; If the data quality of real-time business data and dimension table business data passes the inspection and no risk control rules are triggered, the real-time business data and dimension table business data will be sent to the streaming computing platform.

[0066] In some possible implementations, the data processing device 300 further includes: a routing module; The routing module is used to route real-time business data and dimension table business data to a preset error isolation area if the data quality inspection in real-time business data and dimension table business data fails. Record error details of data routed to the preset error isolation zone and output data quality alarm information.

[0067] In some possible implementations, after outputting data quality alarm information, the data processing device 300 further includes: a creation module; The module is used to create repair tasks for data routed to a preset error isolation zone based on error details; In response to receiving a data repair instruction for a repair task, the data routed to the preset error isolation area is repaired to obtain the repaired data; Based on the repaired data, perform quality verification and risk control rule matching on the data in real-time business data and dimension table business data.

[0068] In some possible implementations, the routing module is also used for: When data in real-time business data and dimension table business data triggers risk control rules, the real-time business data and dimension table business data will be routed to the preset risk control event area. Based on the risk event level of the data routed to the preset risk control event zone, generate risk control alerts corresponding to the risk event level.

[0069] In some possible implementations, the sending module 330 is also used for: Get the topic of the message middleware; Based on the relationship between the topic of the business result data and the topic of the message middleware, select the message middleware that corresponds to the topic of the business result data; The business results data is published to the corresponding message middleware so that data subscribers can obtain the corresponding business results data according to the topic.

[0070] In some possible implementations, the sending module 330 is also used for: Data traffic corresponding to the detection business results data; When the data traffic exceeds a preset threshold, the business results data is stored in the preset data encapsulation interface of the streaming computing platform, and a reminder signal is sent to the data subscription demander to remind the data subscription demander to improve data consumption efficiency.

[0071] The various modules of the data processing apparatus provided in the embodiments of this application can achieve Figure 2 It provides the functionality for each step of the data processing method and enables it to achieve the corresponding technical effects. For the sake of brevity, it will not be elaborated here.

[0072] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0073] The electronic device 400 may include a processor 401 and a memory 402 storing computer program instructions.

[0074] Specifically, the processor 401 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0075] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 402 is non-volatile solid-state memory.

[0076] In certain embodiments, the memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Thus, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0077] The processor 401 implements any of the data processing methods described in the above embodiments by reading and executing computer program instructions stored in the memory 402.

[0078] In some examples, electronic device 400 may also include a communication interface 403 and a bus 404. For example, Figure 4 As shown, the processor 401, memory 402, and communication interface 403 are connected through bus 404 and complete communication with each other.

[0079] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0080] Bus 404 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not as a limitation, bus 404 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 404 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0081] For example, the electronic device 400 can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc.

[0082] The electronic device 400 can execute the data processing method in the embodiments of this application, thereby achieving the combination Figure 2 Describe the data processing methods.

[0083] In addition, in conjunction with the data processing methods in the above embodiments, this application also provides a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the data processing methods in the above embodiments. Examples of computer-readable storage media include non-transitory computer-readable storage media, such as portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, etc.

[0084] This application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, they implement any of the data processing methods described in the above embodiments.

[0085] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0086] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0087] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method is applied to a data processing system, which includes: a database acquisition component, a streaming computing platform, a message middleware connected to the streaming computing platform for message communication, and a data subscription demander; the method includes: In response to a business operation that detects a data change in the upstream business system, business data is collected from the database logs through the database acquisition component; based on the update frequency of the business data, the business data is divided into real-time business data and dimension table business data; the real-time business data is data whose update frequency is greater than a preset threshold, and the dimension table business data is data whose update frequency is not greater than a preset threshold; the real-time business data and the dimension table business data are sent to the streaming computing platform through the database acquisition component; The dimension table business data is stored in a preset initial incremental dimension table cache through the streaming computing platform to obtain an updated incremental dimension table cache. Based on the business identification information, the real-time business data is associated with the dimension table business data in the updated incremental dimension table cache through the streaming computing platform to obtain business result data; The streaming computing platform sends the business result data to the data subscription demander through the message middleware.

2. The data processing method according to claim 1, characterized in that, Before sending the real-time business data and the dimension table business data to the streaming computing platform, the method further includes: The data in the real-time business data and the dimension table business data are subjected to quality verification and risk control rule matching; If the data quality checks of the real-time business data and the dimension table business data pass and no risk control rules are triggered, the real-time business data and the dimension table business data are sent to the streaming computing platform.

3. The data processing method according to claim 2, characterized in that, The method further includes: If the data quality check of the real-time business data and the dimension table business data fails, the real-time business data and the dimension table business data will be routed to a preset error isolation area. Record error details of data routed to the preset error isolation zone and output data quality alarm information.

4. The data processing method according to claim 3, characterized in that, After outputting the data quality alarm information, the method further includes: Based on the error details, a repair task is created for the data routed to the preset error isolation zone; In response to receiving a data repair instruction for the repair task, the data routed to the preset error isolation area is repaired to obtain the repaired data; Based on the repaired data, the quality verification and risk control rule matching of the data in the real-time business data and the dimension table business data are performed.

5. The data processing method according to claim 2, characterized in that, The method further includes: If the risk control rule is triggered by data in the real-time business data and the dimension table business data, the real-time business data and the dimension table business data will be routed to a preset risk control event area. Based on the risk event level of the data routed to the preset risk control event zone, a risk control alarm corresponding to the risk event level is generated.

6. The data processing method according to any one of claims 1-5, characterized in that, The streaming computing platform sends the business result data to the data subscription requester through the message middleware, including: Obtain the topic of the message middleware; Based on the association between the topic of the business result data and the topic of the message middleware, select the message middleware corresponding to the topic of the business result data; The business result data is published to the corresponding message middleware so that the data subscription demander can obtain the corresponding business result data according to the topic.

7. The data processing method according to any one of claims 1-5, characterized in that, The streaming computing platform sends the business result data to the data subscription requester through the message middleware, including: Detect the data traffic corresponding to the business result data; If the data flow exceeds a preset threshold, the business result data is stored in the preset data encapsulation interface of the streaming computing platform, and a reminder signal is sent to the data subscription demander to remind the data subscription demander to improve data consumption efficiency.

8. A data processing apparatus, characterized in that, The device is applied to a data processing system, which includes: a database acquisition component, a streaming computing platform, a message middleware connected to the streaming computing platform for message communication, and a data subscription demander; the device includes: The acquisition module is used to acquire business data from the database logs through the database acquisition component in response to a business operation that detects a data change in the upstream business system; the segmentation module is used to segment the business data into real-time business data and dimension table business data based on the update frequency of the business data; the real-time business data is data whose update frequency is greater than a preset threshold, and the dimension table business data is data whose update frequency is not greater than a preset threshold; the sending module is used to send the real-time business data and the dimension table business data to the streaming computing platform through the database acquisition component; The storage module is used to store the dimension table business data in a preset initial incremental dimension table cache through the streaming computing platform to obtain an updated incremental dimension table cache. The association module is used to associate the real-time business data with the dimension table business data in the updated incremental dimension table cache based on the business identification information through the streaming computing platform to obtain business result data; The sending module is also used by the streaming computing platform to send the business result data to the data subscription demander through the message middleware.

9. A data processing device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the data processing method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1-7.

11. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the data processing method as described in any one of claims 1-7.

Images