Customer asset data processing method and device and electronic equipment

Abstract

The invention discloses a client asset data processing method and device and electronic equipment. The method comprises the following steps: acquiring sub-business asset data corresponding to a target object in a plurality of sub-business systems of a bank; generating sub-business asset theme data corresponding to the first incremental data; determining target theme data from the sub-business asset theme data; determining matching data corresponding to the target theme data from other target theme data corresponding to other sub-service systems; and generating target asset data by using the target theme data and the matching data. According to the method and the device, the technical problem that the customer asset value is inaccurate due to delay, interruption and the like of data transmission when the customer asset data is updated in the related technology under the condition that account exchange exists among the sub-business systems of the bank customer is solved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and in particular, to a method, apparatus, and electronic device for processing customer asset data. Background Art

[0002] In the context of the digital economy era, the trend of bank customers going online is irreversible, and the acquisition of customers' real-time assets plays an important role. The real-time data model aggregation method adopted by related technologies collects the data changes of each source sub-business system separately, and directly pushes the processed data of each subclass of assets to the application end after processing the changed data. However, the transmission of changed data in each source sub-business system may be interrupted or delayed, resulting in inaccurate real-time asset values.

[0003] In response to the above problems, no effective solution has been proposed yet. Summary of the Invention

[0004] Embodiments of this application provide a method, apparatus, and electronic device for processing customer asset data, so as to at least solve the technical problem that in the case of accounting transactions between various sub-business systems of bank customers, the related technologies may cause inaccurate customer asset values due to data transmission delays, interruptions, etc. when updating customer asset data.

[0005] According to one aspect of the embodiments of this application, a method for processing customer asset data is provided, including: obtaining sub-business asset data respectively corresponding to a target object in multiple sub-business systems of a bank, where the sub-business asset data includes first incremental data generated after a preset time point; generating sub-business asset theme data corresponding to the first incremental data, where the sub-business asset theme data is used to represent the asset data of different sub-business systems; determining target theme data from the sub-business asset theme data, where the target theme data includes data generated by transactions between multiple sub-business systems; determining matching data corresponding to the target theme data from other target theme data corresponding to other sub-business systems, where the other sub-business systems are sub-business systems other than the sub-business system corresponding to the target theme data among the multiple sub-business systems, and the matching data and the target theme data together form a complete transaction record; generating target asset data by using the target theme data and the matching data, where the target asset data is used to be sent to the application layer.

[0006] In some embodiments of the present application, determining matching data corresponding to the target theme data from other target theme data corresponding to other sub-business systems includes: obtaining the serial number field value of the target theme data; traversing the first queue to determine whether there is other target theme data with the same serial number field value, where the first queue is used to store target theme data that has not been sent to the application layer; when there is a first target theme data with the same serial number field value, determining the first target theme data as the matching data, and updating the first queue with the target theme data and the first target theme data; when there is no other target theme data with the same serial number field value, determining that there is no matching data corresponding to the target theme data in the first queue, and adding the target theme data to the first queue.

[0007] In some embodiments of the present application, updating the first queue with the target theme data and the first target theme data includes: determining whether the target theme data is core data, where the core data includes data from the core sub-business system; when the target theme data is core data, deleting the matching data in the first queue, and adding the balanced data obtained by combining the target theme data and the matching data to the first queue; when the target theme data is non-core data, updating the matching data to the balanced data obtained by combining the target theme data and the matching data.

[0008] In some embodiments of the present application, before determining that there is no matching data corresponding to the target theme data in the first queue and adding the target theme data to the first queue, the method further includes: traversing the second queue to determine whether there is data corresponding to the target theme data in the second queue, where the second queue is used to store target theme data that has been sent to the application layer; when there is data corresponding to the target theme data in the second queue, determining the target theme data as non-target theme data, and adding the non-target theme data to the first queue.

[0009] In some embodiments of the present application, after determining that there is no matching data corresponding to the target theme data in the first queue and adding the target theme data to the first queue, the method further includes: obtaining the waiting duration for the target theme data to be matched; comparing the duration with a preset duration, and when the comparison result indicates that the duration is greater than the preset duration, sending the target theme data to the application layer.

[0010] In some embodiments of the present application, generating sub-business asset theme data corresponding to first incremental data includes: obtaining the full amount of data of the target object collected at a preset time point, where the full amount of data includes all transaction records before the preset time point corresponding to multiple sub-business systems respectively; constructing first sub-business asset theme data corresponding to multiple sub-business systems respectively by using the full amount of data, where the first sub-business asset theme data includes multiple indicators corresponding to the sub-business system; updating the corresponding indicators in the first sub-business asset theme data by using the first incremental data to obtain the sub-business asset theme data.

[0011] In some embodiments of the present application, the method further includes: obtaining second incremental data generated between the previous preset time point of the preset time point and the preset time point; comparing the second incremental data with the full amount of data to determine whether there is missing data in the second incremental data; in the case where there is missing data in the second incremental data, extracting target data corresponding to the missing data from the full amount of data and determining the target data as the first incremental data.

[0012] According to another aspect of the embodiments of the present application, there is also provided a device for processing customer asset data, including: an obtaining module, configured to obtain sub-business asset data corresponding to a target object in multiple sub-business systems of a bank, where the sub-business asset data includes first incremental data generated after a preset time point; a processing module, configured to generate sub-business asset theme data corresponding to the first incremental data, where the sub-business asset theme data is used to represent the asset data of different sub-business systems; a determining module, configured to determine target theme data from the sub-business asset theme data, where the target theme data includes data generated by transactions between multiple sub-business systems; a matching module, configured to determine matching data corresponding to the target theme data from other target theme data corresponding to other sub-business systems, where the other sub-business systems are sub-business systems other than the sub-business system corresponding to the target theme data among the multiple sub-business systems, and the matching data and the target theme data together form a complete transaction record; an execution module, configured to generate target asset data by using the target theme data and the matching data, where the target asset data is used to be sent to the application layer.

[0013] According to yet another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory and a processor, where the memory is used to store program instructions; the processor is connected to the memory and is configured to execute the method for processing customer asset data as described above.

[0014] According to yet another aspect of the embodiments of the present application, there is also provided a non-volatile storage medium, which includes a stored computer program, where the device where the non-volatile storage medium is located executes the method for processing customer asset data as described above by running the computer program.

[0015] According to another aspect of the embodiments of the present application, there is also provided a computer program product, including computer instructions, which implement the above-mentioned method for processing customer asset data when executed by a processor.

[0016] In the embodiments of the present application, by adopting the method of data balancing verification, by identifying the incremental data generated by transactions between multiple sub-business systems, using the balancing mechanism to determine the matching data corresponding to the incremental data, and then issuing it to the application layer after determining data balancing, the purpose of ensuring the accuracy and consistency of customer asset data is achieved, thus realizing the technical effect of accurately reflecting the real-time asset status of customers and optimizing the customer experience. Furthermore, it solves the technical problem that when there are account transactions between various sub-business systems of bank customers, the relevant technologies may cause inaccurate customer asset values due to reasons such as data transmission delay and interruption during the update of customer asset data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application, and constitute a part of the present application. The schematic embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0018] Figure 1 is a hardware structure block diagram of a computer terminal for a method of processing customer asset data according to an embodiment of the present application;

[0019] Figure 2 is a flowchart of a method of processing customer asset data according to an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the system data flow direction of a method of processing customer asset data according to an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of the balancing enqueue process of a method of processing customer asset data according to an embodiment of the present application;

[0022] Figure 5 is a schematic diagram of the balancing dequeue process of a method of processing customer asset data according to an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of the structure of a device for processing customer asset data according to an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] To enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this application. Obviously, the described embodiments are only a part of the embodiments of this application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts shall fall within the scope of protection of this application.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of this application described here can be implemented in an order different from those illustrated or described here. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices.

[0026] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained as follows:

[0027] Distributed Data Message Queue (abbreviated as DDMQ): The distributed data message queue is a middleware technology used to process a large number of real-time messages in a distributed system. It allows message producers to send messages to the queue, and message consumers to retrieve and process the messages from the queue, thereby achieving asynchronous communication and buffering of messages.

[0028] Distributed Database System (abbreviated as DBS): Such as HBase, it is a database system used to store and process large-scale data, providing persistent storage and efficient access capabilities for data.

[0029] Distributed Processing Engine Architecture Flink (Apache Flink, abbreviated as Flink): It is a distributed stream processing and batch processing framework that supports real-time data stream processing and has high throughput, low latency, and fault tolerance capabilities.

[0030] Driving Table Data: The driving table (or fact table) is a table in a data warehouse used to store business data and metric data, such as a transaction table, a ledger register table, etc., and is the main object of data processing.

[0031] Dimension Table Data: A dimension table is a table used to store descriptive information in a data warehouse, such as product information, exchange rate information, etc., which is used to enrich and expand data. It is usually joined with a fact table (driving table) to provide more detailed context for data analysis.

[0032] Banks generally include a core business system, product business systems, etc. The core business system is responsible for comprehensive transaction processing of the whole bank's business and accounting, facing multiple channels, and providing business processing services such as deposits, loans, payments, and intermediary services. Product business systems include wealth management systems, fund systems, insurance systems, precious metals, etc., which provide services such as agency sales of various products. And bank customer assets consist of deposits, wealth management, bonds, funds, private placements, precious metals, insurance, etc. Therefore, when banks conduct customer asset aggregation, they often need to perform aggregation and statistical processing on the data of the core business system and each product business system. The data sources of these systems are relatively scattered and the processing logic is complex.

[0033] The related technologies mainly use two methods to aggregate data. One is the aggregation method based on online transaction interfaces, and the other is the quasi-real-time data model aggregation method. The online transaction interface aggregation method is that the application end sends customer asset requests to each sub-business system respectively, obtains the result information returned by each sub-business system respectively, and forms aggregated assets and itemized assets after parsing and reprocessing the results. However, this method cannot achieve data reuse, and in addition, the interface logic is complex, which will be invasive to each source business system under high-concurrency requests; the quasi-real-time data model aggregation method is to monitor the data of each sub-business system in real time respectively. When the data of the sub-business system changes, the changed data is processed to generate new account sub-class assets, and the account assets are re-aggregated. Finally, the updated asset data is pushed to the application end. Although this method will not be invasive to each source business system, the transmission of the changed data of each source sub-business system may be interrupted or delayed, resulting in inaccurate aggregated asset result data.

[0034] To solve the above technical problems, the embodiments of the present application provide corresponding solutions, which are described in detail below.

[0035] The method embodiments for processing customer asset data provided by the embodiments of the present application can be executed on a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal for implementing a method for processing customer asset data is shown. As Figure 1As shown, the computer terminal 10 may include one or more processors (illustrated as 102a, 102b, ……, 102n in the figure) (the processor may include, but is not limited to, processing devices such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may further include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those of ordinary skill in the art can understand that Figure 1 the structure shown is only schematic and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 may further include more or fewer components than Figure 1 shown in Figure 1 or have a different configuration from that shown.

[0036] It should be noted that the above one or more processors and / or other data processing circuits are generally referred to as "data processing circuits" in this document. The data processing circuit may be embodied in software, hardware, firmware, or any combination thereof, in whole or in part. In addition, the data processing circuit may be a single independent processing module, or be incorporated in whole or in part into any one of the other elements in the computer terminal 10. As involved in the embodiments of the present application, the data processing circuit is a processor control (such as the selection of a variable resistor terminal path connected to an interface).

[0037] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method of processing customer asset data in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, implements the above-mentioned method of processing customer asset data. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory 104 may further include a memory remotely set relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of the above network include, but are not limited to, the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0038] The transmission module 106 is used to receive or send data via a network. Specific examples of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a Radio Frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0039] The display can be, for example, a touch-screen liquid crystal display (LCD), which enables a user to interact with the user interface of the computer terminal 10.

[0040] It should be noted here that, in some alternative embodiments, the above-mentioned Figure 1 illustrated computer terminal may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware elements and software elements. It should be pointed out that Figure 1 is only an example of a specific specific instance and is intended to illustrate the types of components that may exist in the above-mentioned computer terminal.

[0041] Under the above operating environment, an embodiment of a method for processing customer asset data is provided in the embodiments of the present application. It should be noted that the steps illustrated in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is illustrated in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0042] Figure 2 is a flowchart of a method for processing customer asset data according to an embodiment of the present application. As Figure 2 shown, the method includes the following steps:

[0043] Step S202, obtaining sub-business asset data respectively corresponding to a target object in multiple sub-business systems of a bank, where the sub-business asset data includes first incremental data generated after a preset time point.

[0044] In the above step S202, the target object refers to a specific bank customer whose asset information needs to be obtained. The multiple sub-business systems of the bank refer to the set of sub-systems within the bank that provide various financial services to customers, including but not limited to the wealth management system, fund system, insurance system, deposit system, bond system, precious metal system, etc. These sub-business systems are independent of each other, but need to be uniformly processed when summarizing data. Sub-business asset data refers to specific asset information related to the target object in multiple sub-business systems of the bank. For example, in the wealth management system, the sub-business asset data may include the purchase amount, current market value, share information, etc. of wealth management products; in the deposit system, it may include information such as deposit balance and account type.

[0045] The preset time point refers to a time threshold set by the system to distinguish data generated before and after this time point. The preset time point can be used to determine the time range of incremental data to be collected. The first incremental data refers to the newly added or changed asset data generated by the target object in each sub-business system after the preset time point, which is used to reflect the customer's recent transaction activities and asset changes in each sub-business system.

[0046] In some embodiments of the present application, the Log-based Change Data Capture technology can be used to obtain incremental data of the database by monitoring the online logs or archived logs of each source system database in real time (changes such as insertions, updates, deletions, etc. made to the data in the database). This method has low latency and does not cause invasiveness to the database, and can capture all data changes; optionally, query-based CDC collection can also be used, which periodically queries the database and captures incremental data by comparison. However, this method has high latency, high pressure on the database, and cannot capture all intermediate data changes. Business scenarios with low requirements for data timeliness and no need for intermediate changes can consider using this method.

[0047] Step S204, generate sub-business asset theme data corresponding to the first incremental data, where the sub-business asset theme data is used to represent the asset data of different sub-business systems.

[0048] In the above step S204, the sub-business asset theme data refers to the summary and processing result of the customer asset information for a specific sub-business system (such as wealth management, funds, insurance, etc.), which may include the current state of the asset and multiple key indicators related to the asset type, such as market value, share, redeemable amount, etc., and is used to comprehensively reflect the customer's asset situation in the sub-business system. For example, the key indicators can be determined from the following indicators:

[0049] (1) Customer identification indicators: used to accurately identify and locate specific customer account information, such as customer number and bank card number.

[0050] (2) Product - related indicators: namely, product code, product name, product net value, etc., which are used to determine the specific product types in the sub - business systems corresponding to customers, such as wealth management, funds, bonds, insurance, etc., so as to further determine the asset indicators related to the product.

[0051] (3) Transaction - related indicators: including transaction code, serial number, front - desk serial number, transaction time, transaction amount, transaction confirmation status, balancing status, etc., which are used to help the system identify the nature and scale of transactions.

[0052] (4) Asset indicators: namely, the specific information of assets, including the current share, balance of assets, etc., and can also include the decrease of deposit balance, the increase of wealth management market value, the change of shares, the adjustment of redeemable shares, the handling of unconfirmed funds, etc., which are directly related to the customer's asset status in different sub - business systems.

[0053] (5) Exchange rate and interest rate indicators: in transactions involving different currencies or interest rate changes, they are indicators used to calculate the converted RMB amount or adjust the asset value.

[0054] In some embodiments of the present application, the dimension table data in HBase and the driving table data in Kafka (Apache Kafka) can be read through the Flink processing engine. Through operations such as data association and calculation, customer deposit asset theme data, customer wealth management position - holding theme data, etc. (i.e., sub - business asset theme data) are generated. These data can include multiple indicators such as the customer's asset balance, transaction history, product holding situation, etc., forming an asset view of the customer in each sub - business system.

[0055] In order to ensure the continuity and consistency of data and provide a comprehensive customer asset view for the bank, the sub - business asset theme data corresponding to the first incremental data can be generated through the following steps: Obtain the full - volume data of the target object collected at a preset time point, where the full - volume data includes all transaction records of multiple sub - business systems respectively before the preset time point; Use the full - volume data to construct the first sub - business asset theme data corresponding to multiple sub - business systems respectively, where the first sub - business asset theme data includes multiple indicators corresponding to the sub - business system; Use the first incremental data to update the corresponding indicators in the first sub - business asset theme data to obtain the sub - business asset theme data.

[0056] The preset time point refers to the time point defined by the system for synchronizing or batch collecting data. The full-volume data refers to the data set collected at the preset time point, covering all transaction records of the target objects (i.e., bank customers) in the sub-business system before that time point. For example, at 0:00 in the early morning of a specified date, the system can collect full-volume data from sub-business systems such as the core deposit system, wealth management system, and fund system, including information such as the balances of all customer accounts, transaction records, and product holdings. After collecting the full-volume data, the data can be stored in a distributed database such as HBase for subsequent data processing and analysis.

[0057] In some embodiments of the present application, regarding full-volume data collection, various tools can be used to export the data of each database table in the source system in the form of a batch file. The information collected can include the full-volume data of original tables such as the customer basic information table, wealth management information table, bond information table, insurance information table, deposit information table, fund information table, private placement information table, and precious metal information table. Taking wealth management information and core deposit information as examples, wealth management information includes the entrusted request transaction table, account request transaction table, fund settlement table, share table, share details table, and wealth management product table; core deposit information includes the sub-ledger table, sub-ledger details table, customer contract information table, product type table, interest rate type table, exchange rate type table, deposit type table, wealth management accounting register table, and fund treasure accounting register table, etc.

[0058] The first sub-business asset theme data refers to the result of in-depth processing of the customer asset information in the full-volume data for each sub-business system (such as the wealth management system), including a series of asset indicators directly related to the sub-business system, such as the market value of positions, shares, redeemable amount, etc. In some embodiments of the present application, based on a distributed processing framework such as Apache Flink, the full-volume data can be processed in real time or in batches. The Flink processing engine interacts with HBase to obtain the data of the database table and determines the customer asset theme data of each sub-business system (i.e., the first sub-business asset theme data). For example, for the wealth management system, the Flink processing engine can calculate indicators such as the market value and shares of the wealth management products held by customers to obtain the wealth management asset theme data.

[0059] After the first incremental data is captured in real time, based on the transaction information in the incremental data (such as transaction type, transaction amount, transaction time, etc.), the corresponding first sub-business asset theme data can be located and relevant indicators can be updated. For example, for a transaction of transferring deposits to wealth management, the balance in the deposit sub-business theme data can be reduced, while the market value of positions in the wealth management sub-business theme data can be increased.

[0060] Step S206: Determine the target theme data from the sub - business asset theme data, where the target theme data includes the data generated by transactions between multiple sub - business systems.

[0061] In the above step S206, for the determination of the target theme data, the target theme data can be determined based on the transaction identifier, that is, by identifying specific transaction identifiers such as transaction codes, transaction serial numbers, transaction types, etc., to determine whether the data involves transactions across sub - business systems. For example, for a purchase transaction of a financial product, a purchase transaction record will be generated in the financial management system, and at the same time, there will be a corresponding fund deduction record in the core deposit system. It can be determined that they are related transactions by matching the transaction identifiers (such as the same fields as the transaction serial number, customer ID, etc.) in these two records, and then it can be judged that the purchase transaction record in the financial management system is the target theme data of the transaction involving multiple sub - business systems.

[0062] In some embodiments of the present application, the target theme data can also be determined based on the business logic relationship, that is, by analyzing the relationship between the data based on the business logic to identify the data involving the fund flow between multiple sub - business systems. For example, it can be analyzed whether the fund purchase operation of a customer in the fund system results in a decrease in funds in the core deposit system, or whether the insurance purchase executed in the insurance system is accompanied by changes in assets in the precious metal system. By constructing a business logic model, it is possible to infer which theme data represents the transaction activities across sub - business systems, thereby determining the target theme data.

[0063] When implementing step S206, the above steps can be executed by the Flink distributed processing engine. It should be noted that the above methods can be used alone or in combination to improve the accuracy and stability of the identification of the target theme data. For example, preliminary screening can be first performed based on the transaction identifier, and then the screening results can be further verified through the analysis of the business logic relationship to ensure that the finally determined target theme data is accurate.

[0064] Step S208: Determine the matching data corresponding to the target theme data from the other target theme data corresponding to other sub - business systems, where the other sub - business systems are the sub - business systems other than the sub - business system corresponding to the target theme data among the multiple sub - business systems, and the matching data and the target theme data together form a complete transaction record.

[0065] In the above step S208, the target theme data and the corresponding matching data may be located in different sub - business systems, but their transactions or accounting processes are interrelated. For example, the target theme data may record the purchase transaction in the financial management system, while the matching data records the corresponding decrease in funds in the deposit system.

[0066] In some embodiments of the present application, matching data can be determined based on a transaction identifier, that is, data matching is performed between different sub-business systems through a transaction identifier (such as a transaction serial number, a transaction code, a front-end serial number, etc.). For example, for a target theme data related to wealth management purchase, the system will search for a corresponding deduction record in the deposit system (other sub-business systems) through the transaction serial number A00001 (example) in it. This record will also contain the serial number A00001. After finding the matching record, the system will mark the data in the deposit system as the matching data of the target theme data; for data without a clear transaction identifier, matching data can also be determined based on time series and business logic analysis. For example, the system can record the asset changes of a certain customer before and after a specific time point (such as after time T). If there is a purchase transaction in the wealth management system at time T, and there is a record of a decrease in funds in the deposit system at time T+Δt (Δt is a reasonable time delay), and the amount of the decrease is the same as the amount of the purchase transaction, the system can determine that these two transactions are related to each other based on the matching of the time series and the amount, and thus determine the record of the decrease in funds in the deposit system as the matching data of the target theme data.

[0067] In order to accurately identify the transaction records that need to be updated and avoid redundancy and errors in the data processing process, the matching data corresponding to the target theme data can also be determined through the following steps: obtain the value of the serial number field of the target theme data; traverse the first queue to determine whether there is other target theme data with the same value as the serial number field value, where the first queue is used to store the target theme data that has not been sent to the application layer; when there is a first target theme data with the same value as the serial number field value, determine the first target theme data as the matching data, and update the first queue with the target theme data and the first target theme data; when there is no other target theme data with the same value as the serial number field value, determine that there is no matching data corresponding to the target theme data in the first queue, and add the target theme data to the first queue.

[0068] The value of the serial number field refers to the unique number used to identify each transaction or operation in the banking system. It exists as a key field in the target theme data and is used for subsequent matching and balancing operations. In some embodiments of the present application, in the data processing flow, when the system receives new sub-business asset data, it can parse the data structure of the sub-business asset data, extract the value of the serial number field, and attach its corresponding serial number field when generating the target theme data subsequently.

[0069] The first queue refers to a temporary storage area for storing the intercepted target topic data that has not been sent to the application layer, which can be implemented through a distributed database. In some embodiments of the present application, when new target topic data is received, the data in the first queue can be scanned in real time to find out if there is data with the same serial number.

[0070] If the first target topic data with the same serial number is found in the first queue, these two pieces of data can be regarded as matching data, and according to the preset calculation logic, the matching data can be updated or balanced to ensure that the data in the first queue is in the latest state; if the traversal of the first queue is completed and the first target topic data with the same serial number is not found, the new target topic data can be added to the first queue, that is, the current target topic data has not found its paired transaction or operation and needs to wait for the data in the first queue to be balanced or timeout detected.

[0071] In some embodiments of the present application, when there is the first target topic data with the same value in the serial number field in the first queue, the following steps can be executed to update the first queue by using the target topic data and the first target topic data. Specifically: judge whether the target topic data is core data, where the core data includes data from the core sub-business system; in the case that the target topic data is core data, delete the matching data in the first queue and add the balanced data obtained by combining the target topic data and the matching data to the first queue; in the case that the target topic data is non-core data, update the matching data to the balanced data obtained by combining the target topic data and the matching data.

[0072] Core data refers to the data from the bank's core sub-business system. The core sub-business system is usually responsible for basic services such as accounts and deposits and is the cornerstone of the bank's data processing. In some embodiments of the present application, it can be judged whether the data is core data through specific fields (such as "system source" or "business type" fields) included in the target topic data. In the case that the field value indicates that the data comes from the core system, it is marked as core data; otherwise, it is marked as non-core data.

[0073] Due to its particularity, core data usually has a high authority in the system. Therefore, in the case that the target topic data is core data, the matching data in the first queue can be deleted first and the balanced data obtained by combining the target topic data and the matching data can be added to avoid repeated calculation of the same transaction in subsequent processing and reduce data redundancy. For non-core data, the matching data can be directly updated to the balanced data to avoid unnecessary data deletion and reinsertion operations and simplify the data processing flow. Through this update operation, the data in the first queue can be sorted according to the generation time of the core data to ensure the order of data distribution.

[0074] In order to effectively distinguish the data to be processed from the data that has been processed, avoid duplicate processing and data chaos, and ensure the accuracy and consistency of customer asset data, before adding the target theme data to the first queue, the following steps can also be executed: Traverse the second queue to determine whether there is data corresponding to the target theme data in the second queue, where the second queue is used to store the target theme data that has been sent to the application layer; when there is data corresponding to the target theme data in the second queue, determine that the target theme data is non-target theme data, and add the non-target theme data to the first queue.

[0075] The second queue is used to store the target theme data that has been sent to the application layer and is the output queue after the data in the first queue is processed.

[0076] When there is no first target theme data in the first queue with a serial number field value equal to that of the target theme data, it is also possible to traverse all data records in the second queue and match them with the target theme data through specific fields (such as serial number, customer ID, etc.) to check whether there are exactly the same records. If matching data is found, that is, there is a record with the same serial number or customer ID in the second queue, then it can be determined that the target theme data has been processed and sent to the application layer. In this case, it can be considered that this piece of target theme data does not need to be balanced and is regarded as balanced data (i.e., non-target theme data), and is added to the end of the first queue to wait for subsequent distribution to the application layer.

[0077] Due to network latency or system failures, matching data may be delayed. For business scenarios with high real-time requirements, in order to ensure the timeliness of data updates, after adding the target theme data to the first queue, the following steps can also be executed: Obtain the duration for which the target theme data waits for matching; compare the duration with a preset duration. When the comparison result indicates that the duration is greater than the preset duration, send the target theme data to the application layer.

[0078] The duration for which it waits for matching refers to the time when the target theme data waits in the first queue to be paired with the matching data. The system can use timestamps to record the moment when the target theme data is added to the first queue, and then calculate the duration for which the data waits for matching by subtracting the addition moment from the current moment. For example, the system adds a target theme data of a wealth management purchase to the first queue at time T, and then obtains the current waiting duration of the data in the set at time T + n (n is any positive number) as n.

[0079] The preset duration refers to a time threshold set by the system to determine whether the target theme data times out while waiting to be balanced in the first queue. If the waiting duration exceeds the preset duration, the system will adopt a specific processing mechanism to avoid the data being permanently locked or affecting the real-time accuracy of customer assets.

[0080] When the waiting matching duration of the target subject data exceeds the preset duration, the system can default that the data cannot find matching data. Therefore, it no longer waits for pairing, regards the target subject data as having been balanced (or determined as non-target subject data), and sends it to the application layer to ensure the timeliness and integrity of customer asset information. After sending the target subject data to the application layer, the target subject data can be deleted from the first queue and added to the second queue. In some embodiments of the present application, the above steps can be executed by the Flink distributed processing engine. By monitoring the data waiting time in the first queue in real time and comparing it with the preset duration, if the data waiting time times out, Flink will trigger a queue update operation.

[0081] Step S210: Generate target asset data by using the target subject data and the matching data, where the target asset data is used to be sent to the application layer.

[0082] In the above step S210, the target asset data refers to the final customer asset data record that has been balanced and is used to be sent to the application layer, and may include the total asset information of the customer (target object) in the bank, such as the comprehensive asset view of sub-services such as deposits, wealth management, funds, and insurance. In some embodiments of the present application, the target subject data and the matching data can be calculated and merged according to business rules (such as transaction type, transaction direction) to generate target asset data including the latest value of the customer's total assets. In addition, the target asset data may also include information such as the reason for the change (such as transaction serial number, transaction time, etc.).

[0083] The application layer refers to multiple components or services in the bank system that are used for customers and within the bank. The application layer uses the processed and balanced target asset data to provide services to customers or support the internal decision-making of the bank. In some embodiments of the present application, after receiving the target asset data, the application layer can integrate it into the customer asset view to provide customers with the latest asset information, or use the target asset data for more refined customer analysis and product recommendation.

[0084] To effectively detect and complement data missing problems and ensure the integrity and continuity of data, the following steps can also be executed: Obtain the second incremental data generated between the previous preset time point and the preset time point from the preset time point; Compare the second incremental data with the full amount of data to determine whether there is missing data in the second incremental data; In the case where there is missing data in the second incremental data, extract the target data corresponding to the missing data from the full amount of data and determine the target data as the first incremental data.

[0085] The previous preset time point refers to the time point adjacent to the preset time point in the full - volume data collection cycle. For example, if the preset time point is 0:00 am today, then the previous preset time point is 0:00 am of the previous day (yesterday).

[0086] The missing data refers to the data that fails to be collected or recorded in real - time in the second incremental data due to various reasons (such as network latency, system failures, etc.). In some embodiments of the present application, the full - volume data collected from the source system can be read from storage devices such as HDFS or HBase at the preset time point (such as 0:00 am today). Meanwhile, the Flink engine is used to read the second incremental data stream in Kafka. Through a preset comparison logic, such as exact matching based on the serial number, the system can identify which transaction records are missing in the second incremental data.

[0087] After determining the missing records in the second incremental data, the complete information of these records can be extracted from the full - volume data as the target data. The target data refers to the data records in the full - volume data corresponding to the missing data, which can be used to complement the missing information in the second incremental data and will be re - labeled and processed as the first incremental data to complement the missing part in the real - time data stream.

[0088] Through the above steps S202 to S210, by adopting the method of data balancing verification, by identifying the incremental data generated by transactions between multiple sub - business systems, using the balancing mechanism to determine the matching data corresponding to the incremental data, and then sending it to the application layer after determining the data balance, the purpose of ensuring the accuracy and consistency of customer asset data is achieved. Thus, the technical effect of accurately reflecting the customer's real - time asset status and optimizing the customer experience is realized. Furthermore, the technical problem that in the case of financial transactions between various sub - business systems of bank customers, the relevant technologies may cause inaccurate customer asset values due to data transmission delays, interruptions, etc. during the update of customer asset data is solved.

[0089] The present application also provides a system for processing customer asset data, including: a collection module, a common processing layer, and a balancing module. In addition, the system may also include a source - adhering layer, a basic layer, an application layer, a temporary layer, and a correction module. Among them, the collection module includes collecting full - volume data and incremental data from a multi - heterogeneous database (source system); other parts except the collection module may include a distributed data message queue and a distributed database device. In addition, the source - adhering layer is used to store the collected original data; the basic layer may also include a basic common module based on the distributed processing engine architecture Flink; the common processing layer may also include a common processing module based on the distributed processing engine architecture Flink; the balancing module may also include a balancing service module based on the distributed processing engine architecture Flink; the temporary layer may also include a fault - tolerance processing module based on the distributed processing engine architecture Flink.

[0090] In some embodiments of the present application, the acquisition module is used to obtain sub-business asset data corresponding to a target object in multiple sub-business systems of a bank, where the sub-business asset data includes first incremental data generated after a preset time point; the common processing layer is used to generate sub-business asset theme data corresponding to the first incremental data, where the sub-business asset theme data is used to represent the asset data of different sub-business systems; the balancing module is used to determine target theme data from the sub-business asset theme data, where the target theme data includes data generated by transactions between multiple sub-business systems; determine matching data corresponding to the target theme data from other target theme data corresponding to other sub-business systems, where the other sub-business systems are sub-business systems other than the sub-business system corresponding to the target theme data among the multiple sub-business systems, and the matching data and the target theme data together form a complete transaction record; generate target asset data by using the target theme data and the matching data, where the target asset data is used to be sent to the application layer.

[0091] Figure 3 It is a schematic diagram of the system data flow direction of a method for processing customer asset data according to an embodiment of the present application. As Figure 3 shown, in some embodiments of the present application, the acquisition module collects information from each source business system (such as insurance system, core system, wealth management system, fund system, bond system, etc.), mainly including original full-volume data such as customer basic information, wealth management information, bond information, insurance information, deposit information, fund information, private placement information, precious metal information, etc. (corresponding to full-volume data, that is, data collected in daily frequency batches) and real-time incremental change data (corresponding to incremental data, that is, data collected in real-time increments).

[0092] The source-attached layer receives the data information collected by the acquisition module and retains the original business data and structure to provide simple original appearance access based on the source system structure. Among them, the real-time incremental data is stored in a distributed message queue device, and the full-volume data is stored in a distributed database device, for example, the distributed database device HBase and the distributed file system HDFS.

[0093] The basic layer can perform data shunting and data format standardization processing on the source-attached layer data to form basic layer data. The storage device for the basic layer data can be a distributed message queue device or a distributed database device. Dimension table data such as product tables and exchange rate tables are stored in the Hbase component device, and the incremental data of the driving table such as the water table is written into the distributed message queue device. In addition, all data details will be synchronously written into the distributed file system. The basic layer will transfer the data with processing failures to the temporary layer for subsequent fault tolerance processing.

[0094] The common processing layer performs processing such as association and calculation on the data of the basic layer to form the data of the common processing layer (corresponding to the sub-business asset theme data). The data storage device of the common processing layer can be a distributed message queue device or a distributed database device. It mainly stores various sub-business theme data such as customer wealth holding index data, customer deposit index data, and customer bond holding theme data. The common processing layer will transfer the data with processing failures to the temporary layer for subsequent fault tolerance processing.

[0095] The balancing module performs balancing processing on the sub-business asset theme data of the common processing layer. When the data reconciliation is incomplete, information interception is performed and waiting for balancing. Only after it has been balanced or timed out and no longer requires balancing, will the customer asset data be updated and summarized and sent down to the application layer.

[0096] The application layer receives and stores the customer aggregated asset data and customer itemized asset data transferred by the balancing module, and provides service interfaces for each application to access.

[0097] The temporary layer is used to process abnormal data distribution. The data that the basic layer module, common processing layer module, etc. fail to process normally is stored in the temporary layer. The temporary layer module can distribute the data in the temporary layer to the corresponding basic layer and common processing layer according to the pre-set fault tolerance mechanism at regular intervals, so that the corresponding module can re-process the data.

[0098] The correction module checks and makes up for the real-time incremental data of the source-attached layer. Due to reasons such as network anomalies and unstable collection services, there may be missing real-time incremental data. The source-attached layer can receive incremental data in real time and receive the full-day batch data (corresponding to the full data) of the previous day at midnight every day. The missing real-time incremental data can be corrected with a delay through the full-day batch data, and the corrected incremental data is written into the corresponding basic layer device according to the specification.

[0099] To facilitate understanding of the implementation process of the above embodiments, based on the system for processing customer asset data shown above Figure 3 the following will be explained in combination with a specific embodiment.

[0100] Step S1: The acquisition module collects real-time incremental data (i.e., the first incremental data) and daily frequency batch data (i.e., the full data) from each source business system, and sends them to the source-attached layer.

[0101] Regarding real-time incremental data collection, the log-based CDC incremental data acquisition technology (Log-based Change Data Capture) can be adopted to monitor the online logs or archived logs of each source system database in real time and obtain the incremental database data (such as insertions, updates, deletions, etc. made to the data in the database). Regarding daily batch data collection, various tools can be used to export the data of each database table in the source system in the form of a batch file every day.

[0102] Step S2: The source-attached layer receives and stores the real-time incremental data and daily batch data information of each table collected by the collection module. It retains the original business data and structure and can provide a simple original appearance access based on the source system table structure.

[0103] The real-time incremental data can be stored in the Kafka distributed message queue. Kafka is a distributed, high-throughput message queue system that provides reliable message delivery and persistence functions. It mainly solves problems such as application coupling, asynchronous messages, and traffic peak shaving, and has characteristics such as high performance, high availability, scalability, and eventual consistency, making it very suitable for processing large-scale data stream scenarios. Optionally, components such as RabbitMQ, ActiveMQ, RocketMQ, and Pulsar can also be used as distributed message queues. In Kafka, messages are stored classified by topic (Topic). To standardize the use of components, it is agreed in the source-attached layer that one topic stores the incremental data of one source system table, and each source-attached layer incremental topic data is named according to "O_three-letter abbreviation of the source system_source system table name". Optionally, other naming specifications can also be adopted.

[0104] The full-volume data can be stored in distributed database devices. To standardize the use of components, it can be agreed in the source-attached layer that one table stores the batch data of one source system table, and each source-attached layer batch table data is named according to "O_three-letter abbreviation of the source system_source system table name". Optionally, it can also be stored in devices such as the HDFS (Hadoop Distributed File System) distributed file system, and other naming specifications can also be adopted.

[0105] Step S3: The basic layer performs data shunting and standardization processing on the data in the source-attached layer to form basic layer data, and stores them in different devices according to different data types. The data that fails to be processed during the whole process will be passed to the temporary layer for subsequent fault tolerance processing.

[0106] The basic layer can use the Flink distributed processing engine framework to process the data from the source layer. The Flink processing engine interacts with the Kafka source layer to obtain the data stored in the queue, performs real-time cleaning, standardization, etc., and finally writes the data to different storage media in the basic layer according to the data type. Optionally, other distributed processing engines and frameworks such as Apache Storm and Apache Spark Streaming can also be used.

[0107] The data in the basic layer is mainly divided into dimension table data and driving table data. Dimension table data belongs to auxiliary tables and cannot be used as the main driving table. It is usually used in Join connection scenarios to enrich and expand data. For example, product information tables, exchange rate tables, interest rate tables, and deposit type information tables all belong to dimension table data. Driving table data belongs to the main table and is usually used to drive calculations. For example, transaction tables and accounting register tables all belong to driving table data.

[0108] The dimension table data in the basic layer is stored in the Hbase cluster built on Hadoop and HDFS. To standardize the use of components, the dimension table data in the basic layer can be named according to "F_three-letter abbreviation of the source system_source system table name_business primary key". Optionally, it can also be stored in other storage media that support high-concurrency and low-latency access to massive data, and other naming specifications can also be adopted. The driving table data in the basic layer is stored in the Kafka distributed message queue cluster. Optionally, other distributed message queues can also be used. To standardize the use of components, the topic data in the basic layer can be named according to "F_three-letter abbreviation of the source system_source system table name", and other naming specifications can also be adopted optionally.

[0109] Step S4: The common processing layer performs association, calculation, and other processing on the data from the source layer and the basic layer to form the data of the common processing layer (corresponding to the sub-business asset topic data). The data that fails to be processed during the whole process will be passed to the temporary layer for subsequent fault tolerance processing.

[0110] The common processing layer uses the Flink distributed processing engine framework to process the data from the source layer and the basic layer. The Flink processing engine interacts with Kafka to obtain the driving table data stored in the source layer and the basic layer, and interacts with Hbase to obtain the dimension table data, performs pre-connection, calculation, aggregation, etc., and finally generates the model data of the common processing layer. Optionally, other distributed processing engine frameworks can also be used.

[0111] The model data of the common processing layer will be stored in the Kafka distributed message queue. To standardize the use of components, the data of each topic in the common processing layer can be named according to "C-business abbreviation-business theme". Optionally, it can also be stored in other distributed message queues and other naming specifications can be adopted. The model data of the common processing layer mainly includes the sub-business asset theme data such as customer deposit asset index data, customer wealth management position asset index data, customer fund asset position index data, and customer bond asset position theme data.

[0112] Taking the customer deposit theme data as an example, the main information includes the core customer number, bank card number, product code, product name, main account number, sub-account number, account number, current balance, balance direction, yesterday's balance, deposit type, deposit period, currency, converted RMB amount, serial number (the serial number that causes the update of this index data), front-end transaction code (the product service upload transaction code that causes the update of this index data), front-end serial number (the product service serial number that causes the update of this index data), etc.

[0113] Taking the customer wealth management position theme data as an example, the main information includes the core customer number, bank card number, product code, product name, position market value, position share, redeemable share, non-redeemable share, unconfirmed purchase funds, uncredited redeemed confirmed funds, redeemed share, in-transit amount, product net value, product serial number (the serial number that causes the update of this index data), etc.

[0114] Step S5: The balancing module is responsible for balancing the sub-business asset theme data of the common processing layer. For the data with accounting transactions between systems (corresponding to the target theme data), when the data reconciliation is incomplete, information interception is performed and waiting for balancing. Only after complete balancing or timeout when no balancing is required, will the customer asset data be updated and summarized and sent to the application layer to ensure the accuracy of the asset value.

[0115] Ideally, the total customer asset data is the summary of the sub-business customer asset theme data in the common processing layer. However, these sub-business customer asset theme data come from different sub-business systems and may have accounting transactions with core deposits in some scenarios, such as scenarios of insurance fund transfer, cancellation, and reversal, and scenarios of bond purchase, unexpired interest payment, and redemption. The data of these sub-business systems are collected and processed independently. When the data collection and transmission of each system are interrupted or delayed, it may lead to inaccurate sub-business customer asset theme data and summary asset data of customers.

[0116] Take financial management as an example. If a customer purchases a financial management product worth 1,000 yuan, a financial management purchase record is generated in the financial management system, with the product serial number A00001 (example). The core deposit system will correspondingly freeze or deduct the current deposit of 1,000 yuan, with the serial number B00001 (example) and the front desk serial number A00001 (example). For this transaction, the customer's total asset data remains unchanged. If the collection service of the financial management system is abnormal, the customer's asset information in the customer financial management position subject data of the common processing layer will not be changed, and a new customer data will be added to the customer deposit subject data, which is manifested as an increase in assets and will be accompanied by this serial number, product serial number, transaction code and other information. If the customer asset subject data of each sub-business in the common processing layer is directly aggregated, the customer's total assets will increase in disguise. This application sets up a balancing mechanism. In this scenario, based on the transaction code in the customer's deposit subject data, the front-end serial number A00001 and other information, it will be determined that the account transactions are incomplete, and it is necessary to intercept and wait for the customer's wealth management holdings subject data for reconciliation. Until the customer adds a new serial number A00001 in the core wealth management holdings subject data that is the same as the front-end serial number (i.e. matching data), it is determined that the balance has been achieved, and the customer's total asset data will be summarized and pushed to the application end. At this time, the customer's total assets are consistent with before.

[0117] The balancing module includes three submodules: balancing queue entry, balancing queue exit, and balancing timeout detection. These three submodules all use the Flink distributed processing engine framework to balance the sub-business asset topic data of the common processing layer. The Flink processing engine interacts with Kafka to obtain the customer sub-business asset data stored in the common processing layer queue, and interacts with the customer balancing queue information in Hbase to perform balancing queue entry, balancing update, timeout detection, balancing distribution, etc., and finally generates customer asset model data for use on the application side. Optionally, other distributed processing engine frameworks or distributed message queues can also be used.

[0118] The balancing operation mainly processes the balancing queue (corresponding to the first queue) information. The balancing queue information (at least including the target subject data) can be implemented based on "customer number + bank card number". The information can be stored in the Hbase table, with "customer number + bank card number" as the rowkey (row key, the unique identifier of each row of data), and the value (specific data value) is the pending information under the bank card number; the information can also be stored in other high-concurrency and low-latency massive distributed databases, that is, with "customer number + bank card number" as the key, the transaction information that has been queued under the bank card number can be found.

[0119] The balancing enqueue sub-module is responsible for performing the enqueue operation on the target topic data. When enqueuing, it will perform the to-be-balanced enqueue operation or the balanced update enqueue operation according to information such as whether the current target topic data is core data and whether it needs to be balanced. In some embodiments of the present application, the core data can be data generated by a core system (such as a deposit system), and the business data is data generated by sub-business systems other than the core system. The processing flow is divided into the following several types:

[0120] (1) If the data to be enqueued (target topic data) is core data and business data that can be balanced can be found in the corresponding queue, then delete the business data in the queue and insert the balanced data composed of the business data and the core data at the end of the queue, and this enqueue processing ends.

[0121] (2) If the data to be enqueued is core data and business data that can be balanced cannot be found in the corresponding queue. In this case, if the data needs to be balanced and the corresponding information of the business data can be found in the already issued information (the already issued information is stored in the second queue), then update the mark of the data to indicate that it does not need to be balanced, otherwise no update processing is performed. Then insert the data information at the end of the queue, and this enqueue processing ends.

[0122] (3) If the data to be enqueued is business data and core data that can be balanced can be found in the corresponding queue, then update the corresponding core data in the queue to the balanced data composed of the business data and the core data, and this enqueue processing ends.

[0123] (4) If the data to be enqueued is business data and core data that can be balanced cannot be found in the corresponding queue. In this case, if the data needs to be balanced and the corresponding information can be found in the already issued information, then update the mark of the data to indicate that it does not need to be balanced, otherwise no update processing is performed. Then insert the data information at the end of the queue, and this enqueue processing ends.

[0124] Figure 4 It is a schematic diagram of the balancing enqueue process of a method for processing customer asset data according to an embodiment of the present application, as Figure 4 shown

[0125] Step S402: Obtain data, that is, obtain the sub-business asset topic data of the customer from the common processing layer.

[0126] Step S404: Determine whether the data needs to be balanced, that is, determine whether the data needs to be balanced according to the type of the data and the business logic. Among them, data with financial transactions between sub-business systems needs to be balanced (corresponding to the target topic data). If the data does not need to be balanced, execute step S406; if the data needs to be balanced, execute step S412.

[0127] Step S406: Determine whether the data is core data. If the data does not need to be balanced, further determine whether the data is core data. Core data generally refers to data from the core business system, such as transaction data in the deposit system. It can be determined whether the data is core data by the system identification field in the data (such as the three-letter abbreviation of the source system). If the data is core data, execute step S408; if the data is not core data, execute step S418.

[0128] Step S408: Determine whether the data in the queue can be balanced. If the data does not need to be balanced and is core data, check whether there is business data in the queue that can be balanced (i.e., data other than core data). If the data in the queue can be balanced, execute step S410; if the data in the queue cannot be balanced, execute step S418.

[0129] Step S410: Delete the business data and convert it into balanced data. If there is business data in the queue that can be balanced, delete the business data and combine it with the core data to form balanced data. After executing step S410, execute step S418.

[0130] Step S418: Enqueue at the tail, that is, rewrite the data that needs to be processed to the tail of the Hbase balancing queue for continued processing. After executing step S418, execute step S428.

[0131] Step S428: End the process, that is, after the data processing is completed, end the current processing process.

[0132] Step S412: Determine whether the data is core data, that is, if the data needs to be balanced, further determine whether the data is core data. If the data is core data, execute step S414; if the data is not core data, execute step S420.

[0133] Step S414: Determine whether it can be balanced in the queue, that is, if the data needs to be balanced and is core data, check whether there is business data in the queue that can be balanced. If the data in the queue can be balanced, execute step S416; if the data in the queue cannot be balanced, execute step S422.

[0134] Step S416: Delete the business data and convert it into balanced data, that is, if there is business data in the queue that can be balanced, delete the business data and combine it with the core data to form balanced data. The found business data can be deleted from the Hbase queue, and the deleted business data and the core data are combined into new balanced data, and the new balanced data is rewritten to the tail of the Hbase queue (that is, execute step S418).

[0135] Step S422: Find the balanced data in the published C_BALANCE_RESULT in HBase, that is, if the data cannot be balanced in the queue, try to find the balanced data from the published data in HBase. The balanced data can be found from the published data in HBase by using the customer number and bank card number as the RowKey. If the balanced data is found in the published data in HBase, execute Step S424; if the balanced data is not found in the published data in HBase, execute Step S418.

[0136] Step S424: Mark as not requiring balanced data, that is, if the balanced data is found in the published data in HBase, mark the data as not requiring balancing.

[0137] Step S420: Determine whether the data in the queue can be balanced (all core data), that is, if the data needs to be balanced and is not core data, check whether there is core data in the queue that can be balanced. The data at the head of the queue can be read from the hbase balancing queue, and compared with the balancing queue information in HBase through the Flink distributed processing engine to determine whether there is core data that can be balanced. If there is core data in the queue that can be balanced, execute Step S426; if the core data in the queue cannot be balanced, execute Step S422.

[0138] Step S426: Convert the core data in the queue that can be balanced into balanced data, that is, if there is core data in the queue that can be balanced, combine the core data with the business data into balanced data. Then execute Step S428 to end the process.

[0139] It should be noted that the condition for the core data and the business data to be balanceable is that the serial number field values of the core data and the business data are the same. The issued information (i.e., the data that has been used to update the sub-business asset data) can be stored in the Hbase table "C_BALANCE_RESULT", or in other high-concurrency and low-latency massive distributed databases. The data stored in this table can include the issued data that has been balanced, the issued data that does not require balancing, and the timed-out issued data. Due to situations such as abnormal acquisition services, the source layer may receive duplicate incremental data, so both business metrics and deposit metrics may be double-counted, and the balancing module will also process this incremental data twice. If the balancing compatibility for duplicate data is not supported, it will cause a situation where the second processing gets stuck in the queue due to the already balanced issued data in the first processing. Therefore, the balancing compatibility processing needs to determine whether the information has been issued, and if so, mark it as not requiring balancing.

[0140] The balancing dequeue sub-module is responsible for performing data dequeue operations. For data that has been balanced or does not require balancing, this module will perform data dequeue. The dequeued data will be sent down to the application layer Kafka, and at the same time, the sending record will be saved to the Hbase table "C_BALANCE_RESULT". Figure 5 It is a schematic diagram of the balancing dequeue process of a method for processing customer asset data according to an embodiment of the present application, as Figure 5 shown, which includes the following steps:

[0141] Step S502: Traverse the queue. The system starts traversing from the head of the balancing queue until the tail of the queue, and checks the balancing status of each queue element.

[0142] Step S504: Determine whether the element needs to be balanced. Read the balancing flag field in the queue element, or determine whether the data needs to be balanced through business logic. If the data needs to be balanced, jump to step S506; if the data does not need to be balanced, jump to step S508.

[0143] Step S506: Determine whether the element has been balanced. Check whether the queue element has been marked as balanced, or query the HBase table C_BALANCE_RESULT to confirm whether the data has been sent down. If it has been sent down, it is determined that the current data is also considered balanced. If the data has been balanced, jump to step S508; if the data has not been balanced, jump to step S510.

[0144] Step S508: Dequeue. Delete the current element from the balancing queue. Transfer the data to the application layer for updating and summarizing customer asset data, save the processing record in the HBase table C_BALANCE_RESULT, and continue to execute step S502 to continue traversing the first element of the queue.

[0145] Step S510: End the process. Keep the current data in the queue and do not perform dequeue operations. End the execution of the current processing process and wait to check the balancing status of this data again when traversing the queue next time.

[0146] In some embodiments of the present application, the dequeue processing process can be:

[0147] (1) If the balancing flag of the data at the head of the queue does not require balancing, directly send down the data, delete the data at the head of the queue, and continue to judge the data at the head of the queue in the next round.

[0148] (2) If the balancing flag of the data at the head of the queue requires balancing and the balanced data also exists, it means it has been balanced. Dequeue the data, send it down to the specified application layer storage medium, and save the sending record to the Hbase table at the same time. Continue to process the data at the head of the queue in the next round.

[0149] (3) If the data balancing flag at the head of the queue indicates that balancing is required and the balancing data does not exist, it means that the data has not been balanced yet and waiting is needed, thus ending the data dequeue process for balancing.

[0150] The balancing timeout sub-module is responsible for detecting timeout data. It scans all non-empty queues (corresponding to the first queue) at regular intervals for timeout data detection. For each non-empty queue, it determines whether the head element has exceeded the configured timeout. If it has exceeded the set timeout, the data is marked as not requiring balancing (corresponding to the determination of non-target topic data).

[0151] In addition to the above steps, there are also two bypass steps:

[0152] Bypass step 1: The temporary layer mainly processes the distribution of abnormal data. The temporary layer stores data that the basic layer module, common processing layer module, etc. have failed to process normally (such as sub-business asset data, sub-business asset topic data, etc.). The temporary layer module distributes the data in the temporary layer to the corresponding basic layer and common processing layer according to the pre-set fault tolerance mechanism at regular intervals for the corresponding modules to reprocess the data. The temporary layer module uses the Flink distributed processing engine framework to process the data in the temporary layer. The Flink processing engine obtains the driving table data stored in the temporary layer by interacting with Kafka, and redistributes the data in the temporary layer according to the pre-set processing cycle, maximum number of times, and other information. Optionally, other distributed processing engine frameworks can also be used.

[0153] Bypass step 2: The correction module checks and makes up for the missing real-time incremental data in the source-attached layer. The source-attached layer receives the incremental data transmitted by the acquisition module in real time and receives the full batch data of the previous day transmitted by the acquisition module every morning. Due to reasons such as network anomalies and unstable acquisition services, there may be missing real-time incremental data. The missing real-time incremental data can be corrected later using the full batch data. The supplemented and corrected incremental data is written into the corresponding basic layer devices according to the specifications for subsequent processing. The correction module uses the Flink distributed processing engine framework to process the incremental data in the source-attached layer, writes the data into the HDFS component, and compares it with the full data through a program written in Java. The data determined to be missing after comparison is written into the corresponding basic layer devices according to the data type.

[0154] In the above steps, distributed big data processing technology is adopted. First, the real-time incremental data and daily frequency batch data of each sub-business system are collected; second, through standardized cleaning and common processing, a data asset model is formed to achieve data reuse and sharing, avoiding intrusion into each business system; third, through the balancing mechanism, the incompleteness of data reconciliation is intercepted to ensure the accuracy of customer asset values. Through the correction mechanism, the missing data is filled, avoiding manual data filling operations and effectively reducing the maintenance cost.

[0155] Figure 6 is a structural diagram of a device for processing customer asset data according to an embodiment of the present application, as Figure 6 shown. The device includes:

[0156] An acquisition module 602, configured to acquire sub-business asset data corresponding to a target object in multiple sub-business systems of a bank respectively, where the sub-business asset data includes first incremental data generated after a preset time point;

[0157] A processing module 604, configured to generate sub-business asset theme data corresponding to the first incremental data, where the sub-business asset theme data is used to represent asset data of different sub-business systems;

[0158] A determination module 606, configured to determine target theme data from the sub-business asset theme data, where the target theme data includes data generated by transactions between the multiple sub-business systems;

[0159] A matching module 608, configured to determine matching data corresponding to the target theme data from other target theme data corresponding to other sub-business systems, where the other sub-business systems are sub-business systems other than the sub-business system corresponding to the target theme data among the multiple sub-business systems, and the matching data and the target theme data together form a complete transaction record;

[0160] An execution module 610, configured to generate target asset data by using the target theme data and the matching data, where the target asset data is used to be sent to the application layer.

[0161] It should be noted that Figure 6 the device for processing customer asset data shown is used to execute Figure 2 the method for processing customer asset data shown. Therefore Figure 2 the relevant explanations in the method for processing customer asset data in Figure 6 also apply to the device for processing customer asset data shown, which will not be elaborated here.

[0162] An embodiment of the present application further provides an electronic device, which includes a memory and a processor. The memory is used to store program instructions; the processor is connected to the memory and is configured to execute the steps of implementing the method for processing customer asset data in various embodiments of the present application.

[0163] For example, the processor executes the following functions by executing the program instructions stored in the memory:

[0164] Obtain the sub - business asset data corresponding to the target object in multiple sub - business systems of the bank, where the sub - business asset data includes the first incremental data generated after a preset time point; generate sub - business asset theme data corresponding to the first incremental data, where the sub - business asset theme data is used to represent the asset data of different sub - business systems; determine target theme data from the sub - business asset theme data, where the target theme data includes the data generated by transactions between multiple sub - business systems; determine matching data corresponding to the target theme data from other target theme data corresponding to other sub - business systems, where the other sub - business systems are the sub - business systems other than the sub - business system corresponding to the target theme data among the multiple sub - business systems, and the matching data and the target theme data together form a complete transaction record; generate target asset data using the target theme data and the matching data, where the target asset data is used to be sent to the application layer.

[0165] The embodiment of the present application also provides a non - volatile storage medium, which includes a stored computer program. Wherein, the device where the non - volatile storage medium is located executes the steps of the method for processing customer asset data in each embodiment of the present application by running the computer program.

[0166] The embodiment of the present application also provides a computer program product, including computer instructions, which implement the steps of the method for processing customer asset data in each embodiment of the present application when executed by a processor.

[0167] The embodiment of the present application also provides a computer program, which implements the steps of the method for processing customer asset data in each embodiment of the present application when executed by a processor.

[0168] The serial numbers of the above - mentioned embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments.

[0169] In the above - mentioned embodiments of the present application, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0170] In several embodiments provided by the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units can be a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in an electrical or other form.

[0171] The unit described as a separation component may or may not be physically separated. The component shown as a unit may or may not be a physical unit, that is, it may be located in one place or distributed across multiple units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0172] In addition, each functional unit in various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

[0173] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: USB flash drives, read-only memories (ROMs), random access memories (RAMs), mobile hard disks, magnetic disks, or optical discs and other various media that can store program codes.

[0174] The above are only the preferred embodiments of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for processing customer asset data, characterized in that: include: Acquire sub-business asset data corresponding to the target object in multiple sub-business systems of the bank, wherein the sub-business asset data includes first incremental data generated after a preset time point; Generate sub-business asset subject data corresponding to the first incremental data, wherein the sub-business asset subject data is used to represent asset data of different sub-business systems; Determining target subject data from the sub-business asset subject data, wherein the target subject data includes data generated by transactions between the multiple sub-business systems; Determining matching data corresponding to the target subject data from other target subject data corresponding to other sub-business systems, wherein the other sub-business systems are sub-business systems other than the sub-business system corresponding to the target subject data among the multiple sub-business systems, and the matching data and the target subject data together constitute a complete transaction record; The target subject data and the matching data are used to generate target asset data, wherein the target asset data is used to be sent to an application layer.

2. The method according to claim 1, characterized in that Determining matching data corresponding to the target subject data from other target subject data corresponding to other sub-business systems includes: Obtain the serial number field value of the target subject data; Traversing the first queue to determine whether there is other target subject data equal to the serial number field value, wherein the first queue is used to store the target subject data that is not sent to the application layer; When there is first target subject data having a value equal to the serial number field value, determining that the first target subject data is the matching data, and updating the first queue using the target subject data and the first target subject data; When there is no other target subject data with the same value as the serial number field, it is determined that there is no matching data corresponding to the target subject data in the first queue, and the target subject data is added to the first queue.

3. The method according to claim 2, characterized in that Updating the first queue using the target subject data and the first target subject data includes: Determining whether the target subject data is core data, wherein the core data includes data from a core sub-business system; In the case where the target subject data is core data, deleting the matching data in the first queue, and adding balancing data obtained by combining the target subject data and the matching data into the first queue; In the case that the target subject data is non-core data, the matching data is updated to balanced data obtained by combining the target subject data with the matching data.

4. The method according to claim 2, characterized in that: Before determining that there is no matching data corresponding to the target subject data in the first queue and adding the target subject data to the first queue, the method further includes: Traversing the second queue to determine whether there is data corresponding to the target subject data in the second queue, wherein the second queue is used to store the target subject data that has been sent to the application layer; When data corresponding to the target subject data exists in the second queue, the target subject data is determined to be non-target subject data, and the non-target subject data is added to the first queue.

5. The method according to claim 2, characterized in that: After determining that there is no matching data corresponding to the target subject data in the first queue, and adding the target subject data to the first queue, the method further includes: Obtain the length of time the target subject data waits for matching; The duration is compared with a preset duration, and when the comparison result indicates that the duration is greater than the preset duration, the target subject data is sent to the application layer.

6. The method according to claim 1, characterized in that Generating sub-business asset subject data corresponding to the first incremental data includes: Acquire the full amount of data of the target object collected at the preset time point, wherein the full amount of data includes all transaction records before the preset time point corresponding to the multiple sub-business systems respectively; Using the full data, construct first sub-business asset subject data corresponding to the multiple sub-business systems respectively, wherein the first sub-business asset subject data includes multiple indicators corresponding to the sub-business systems; The first incremental data is used to update the corresponding index in the first sub-business asset subject data to obtain the sub-business asset subject data.

7. The method according to claim 6, characterized in that The method further comprises: Acquire second incremental data generated from a previous preset time point of the preset time point to the preset time point; Comparing the second incremental data with the full data to determine whether there is missing data in the second incremental data; In the case where there is missing data in the second incremental data, target data corresponding to the missing data is extracted from the full data, and the target data is determined to be the first incremental data.

8. A device for processing customer asset data, characterized in that: include: An acquisition module, used to acquire sub-business asset data corresponding to the target object in multiple sub-business systems of the bank, wherein the sub-business asset data includes first incremental data generated after a preset time point; A processing module, used to generate sub-business asset subject data corresponding to the first incremental data, wherein the sub-business asset subject data is used to represent asset data of different sub-business systems; A determination module, configured to determine target subject data from the sub-business asset subject data, wherein the target subject data includes data generated by transactions between the multiple sub-business systems; A matching module, used to determine matching data corresponding to the target subject data from other target subject data corresponding to other sub-business systems, wherein the other sub-business systems are sub-business systems other than the sub-business system corresponding to the target subject data among the multiple sub-business systems, and the matching data and the target subject data together constitute a complete transaction record; An execution module is used to generate target asset data using the target subject data and the matching data, wherein the target asset data is used to be sent to an application layer.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the method for processing customer asset data as described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the method for processing customer asset data as described in any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method for processing customer asset data described in any one of claims 1 to 7 is implemented.

Images