Data management system and method, terminal equipment and storage medium

Through the hierarchical processing of the data management system and the storage strategies of different levels of data, the problems of data redundancy and low query efficiency in banking business are solved, efficient data management and system collaboration are achieved, and rapidly changing business needs are adapted to.

CN119066093AInactive Publication Date: 2024-12-03KINCHENG BANK OF TIANJIN CO LTD

Patent Information

Application Number
CN202411563803.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-05
Publication Date
2024-12-03
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

In banking business processing, data storage and management complexity increases, data redundancy, query efficiency is low in traditional database mode, and the coupling between systems is high, making it difficult to adapt to rapidly changing business needs.

Method used

The data management system is adopted to classify the data through the business logic layer, adaptation layer and interface interaction layer, and divide the data into hot data, temperature data and cold data, and use message queue, incremental synchronization and regular archive strategies respectively. Storage and synchronization, optimize queries and updates using cache and lock mechanisms.

Benefits of technology

It improves data query efficiency, reduces system maintenance costs, enhances the flexibility and coordination efficiency of the data management system, ensures data consistency and real-timeness, and supports rapid response to business changes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066093A_ABST
    Figure CN119066093A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, and discloses a data management system and method, terminal equipment and a storage medium, and the system comprises a business logic layer, an adaptation layer and an interface interaction layer; the business logic layer comprises an account model for identifying account information in the data and a user model for identifying user information in the data; the interface interaction layer is connected with different memories and different target ends; when the storage request is responded, the business logic layer carries out grading processing on user information and / or account information generated by a target end to obtain hot data, temperature data and / or cold data; and the adaptation layer transmits the hot data to the first-class database in a message queue manner, and / or transmits the hot data to the second-class database through an increment synchronization strategy, and / or transmits the cold data to the third-class database in a regular filing manner. According to the data management system, collaborative management is more friendly.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data management system, method, terminal device and storage medium. Background Art

[0002] In the current banking business processing, with the deepening of digital transformation, the amount of bank account data is growing exponentially, which directly leads to a significant increase in the complexity of data storage and management. Traditional bank account management systems mostly rely on relational database technology and adopt multi-database backup strategies to ensure data security and reliability. However, this model has the following problems: 1. In a multi-database environment, banks have complex data replication and synchronization mechanisms, which can easily lead to data redundancy; 2. Faced with massive data, especially the rapid growth of unstructured data, traditional query methods often need to traverse multiple databases or tables, and insufficient index optimization seriously affects the query speed. In addition, under high-concurrency operations, data synchronization cannot be achieved in real time, so the query efficiency is low. 3. Data sharing between business systems within the bank is not smooth, and the coupling between systems is too high. Modifying one or some data may require synchronous update processing in multiple systems, which increases system maintenance costs and makes it difficult to adapt to rapidly changing business needs, that is, coordination between systems is difficult. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a data management system, method, terminal device and storage medium, which can effectively solve the problems of poor query efficiency and difficulty in collaboration in the current bank data management process.

[0004] In a first aspect, an embodiment of the present application provides a data management system, including: a business logic layer, an adaptation layer, and an interface interaction layer; the data includes user information and account information opened by the user; The business logic layer includes an account model and a user model, wherein the user model is used to identify the user information and the account model is used to identify the account information; the interface interaction layer is used to connect different storage devices and different target terminals; When responding to a storage request, the business logic layer performs hierarchical processing on the user information and / or the account information generated by the target end to obtain hot data, warm data and / or cold data; the adaptation layer transmits the obtained hot data through a message queue and transmits it to the first type of database via the interface interaction layer, and / or transmits the obtained warm data through an incremental synchronization strategy and transmits it to the second type of database via the interface interaction layer, and / or transmits the obtained cold data through periodic archiving and transmits it to the third type of database via the interface interaction layer.

[0005] In some embodiments, the data management system further comprises: In response to a read request, the adaptation layer retrieves the hot data from the first type of database, retrieves the warm data from the second type of database, and / or retrieves the cold data from the third type of database through the interface interaction layer, and inputs the retrieved hot data, the warm data, and / or the cold data into the business logic layer. The business logic layer identifies the hot data, the warm data, and / or the cold data through the user model and / or the account model and outputs the corresponding user information and / or account information. The adaptation layer outputs the corresponding user information and / or account information to the corresponding target end through the interface interaction layer.

[0006] In some embodiments, the business logic layer identifies the hot data, which includes: The business logic layer queries the cache area: When the hot data is in the cache area, reading the hot data from the cache area; When the hot data is not in the cache area, the hot data is automatically loaded from the first type of database, and the hot data in the cache area is updated.

[0007] In some embodiments, updating the hot data in the cache includes: The business logic layer triggers cache instructions through a publish / subscribe mode; The adaptation layer uses an optimistic lock or a pessimistic lock strategy to retrieve the hot data from the first type of database according to the cache instruction, and loads the retrieved hot data into the cache area for updating.

[0008] In some embodiments, the adaptation layer includes an input adapter and an output adapter; The input adapter is used to configure the input interface and transmit the data generated according to the user behavior and the hot data, the warm data and / or the cold data retrieved from different databases to the business logic layer; The output adapter is used to configure the output interface, transmit the user information and / or the account information in the business logic layer to different target ends, and transmit the hot data, the warm data and / or the cold data obtained by hierarchical processing to different databases.

[0009] In a second aspect, an embodiment of the present application provides a data management method, which is applicable to the above-mentioned data management system; the data management method comprises: In response to the storage request, the business logic layer performs hierarchical processing on the input data to obtain hot data, warm data and / or cold data; The adaptation layer receives the hot data, the warm data and / or the cold data output by the business logic layer, and transmits the received hot data through a message queue, transmits the received warm data through an incremental synchronization strategy, and / or transmits the received cold data to the interface interaction layer through periodic archiving; The interface interaction layer stores the hot data in the first type of database, stores the warm data in the second type of database, and stores the cold data in the third type of database.

[0010] In some embodiments, the business logic layer processes the data in a graded manner according to the access frequency and importance level of the data to obtain the hot data, the warm data and / or the cold data respectively.

[0011] In a third aspect, an embodiment of the present application provides a data management method, which is applicable to the above-mentioned data management system; the data management method comprises: In response to a read request, when the hot data needs to be read, the adaptation layer first loads the hot data in the first type of database into a cache area, and then the business logic layer accesses and reads the hot data in the cache area; When the warm data needs to be read, the business logic layer accesses the second type of database via the adaptation layer, and reads the warm data from the second type of database; When the cold data needs to be read, the business logic layer accesses the third type of database via the adaptation layer, and reads the cold data from the third type of database; The business logic layer identifies the read hot data, the warm data and / or the cold data, and outputs corresponding user information and / or account information after identification; The adaptation layer outputs the corresponding user information and / or account information to the corresponding target end through the interface interaction layer.

[0012] In a fourth aspect, an embodiment of the present application provides a terminal device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned data management method.

[0013] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the above-mentioned data management method when executed on a processor.

[0014] The embodiments of the present application have the following beneficial effects: The data management system proposed in the present application manages data in a data classification manner to obtain hot data, warm data and cold data of different levels, and adopts different storage strategies and access mechanisms for data of different levels, so as to ensure the response requirements of data of different levels; moreover, the business logic layer of the data management system of the present application is encapsulated in the middle, and the periphery is connected to various external databases and target ends through the adaptation layer and the interface interaction layer; this data management system has good structural flexibility and scalability, and can enable each target end in the data management system to evolve independently relative to the business logic layer, reduce coupling, and thus improve the collaborative efficiency of each target end in the data management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A schematic diagram of the architecture of a data management system according to an embodiment of the present application is shown; Figure 2 A flow chart showing the data management system in accordance with an embodiment of the present application responding to a storage request is shown; Figure 3 A flow chart showing the data management system in accordance with an embodiment of the present application responding to a read request is shown; Figure 4 A first flow chart of the data management method according to an embodiment of the present application is shown; Figure 5 A second flow chart of the data management method according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0018] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0019] Hereinafter, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0020] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0021] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0022] In the process of storing and managing massive amounts of data in a multi-database environment, a large amount of backup and redundant storage not only occupies a large amount of hardware resources, but also increases the complexity and cost of operation and maintenance management; for data-intensive industries such as banks, data synchronization, query efficiency, and inter-system synergy play an important role. Therefore, the present application proposes a management system, method, terminal device, and storage medium for bank data. The bank data management method aims to improve data processing efficiency and ensure data consistency and real-time performance by classifying the bank's data and using a data management architecture that can be structured with the bank's system.

[0023] The data management system is described below with reference to some specific data in a banking system as an example.

[0024] Figure 1 A structural diagram of a data management system applicable to a bank according to an embodiment of the present application is shown. Exemplarily, the management system includes a business logic layer, an adaptation layer and an interface interaction layer; the data includes user information and account information opened by the user.

[0025] In this application example, the banking system is applicable, so the data specifically includes user information and account information opened by the user in the bank; as other implementation methods, it is also applicable to other financial management systems such as stocks, bonds, etc.

[0026] The business logic layer of this application includes an account model and a user model. The user model is used to identify user information, and the account model is used to identify the account information opened by the user in the bank. The business logic layer is located in the center of the architecture and encapsulates all account-related business rules and logic to ensure the purity of the data without directly relying on any external system or technical details.

[0027] In order to better manage the bank's data and improve the query efficiency during the management process, this application performs hierarchical processing on the bank's data. Specifically, the bank's data is divided into three levels according to the access frequency of the bank's data, namely hot data, warm data and cold data. In this example, hierarchical processing is performed according to the access frequency of the bank's data. As another implementation method, hierarchical processing is performed according to the access frequency and importance of the bank's data. The combination of access frequency and importance has a better effect on the hierarchical processing of the bank's data.

[0028] In this application, hot data includes the latest transaction records and basic user information that are frequently accessed; it can be understood that hot data refers to information that often needs to be quickly queried and processed in daily business operations, usually with high access frequency and low latency requirements. Exemplarily, hot data includes user login information, transaction records within the last month, account balance, real-time exchange rate, etc.; these data are crucial to the user's instant experience and require millisecond query speeds.

[0029] In this application, warm data includes historical transaction records that are accessed regularly; it is understandable that compared with hot data, warm data is accessed less frequently, but it still needs to maintain a certain query efficiency, and is usually used in scenarios such as data analysis, report generation, or customer historical behavior analysis. Exemplarily, warm data includes transaction records within the past year, quarterly financial statements, customer behavior logs for the past 6 months, etc.; although this type of data is not frequently queried every day, it will be used in specific business cycles or analysis needs, requiring both query efficiency and storage costs.

[0030] In this application, cold data includes historical data that is archived for a long time; it is understandable that cold data refers to data that is rarely accessed but must be retained for reasons such as legal compliance and historical record preservation. For example, transaction records over 5 years old, detailed information of closed accounts, historical email archives, etc. Access to such data is sporadic and the access frequency is extremely low; that is, for the data of the bank stored in archives, the focus is more on long-term preservation of data rather than fast access, so the query speed is not high.

[0031] The hot data of the present application is allocated and stored in the first type of database through the message queue, the warm data is allocated and stored in the second type of database through the incremental synchronization strategy, and the cold data is allocated and stored in the third type of database through regular archiving.

[0032] In some embodiments, hot data is crucial to the user's instant experience and requires a fast response, so it is suitable for storage in a first-class database such as an in-memory database and / or a high-performance cache to ensure a millisecond query speed. In some embodiments, hot data is stored in Redis, which is an open source, network-supported, memory-based, and optionally persistent key-value pair storage database written in ANSI C. It contains multiple data structures and supports networking. It is one of the most popular NoSQL databases.

[0033] The hot data of this application is synchronously managed by means of message queues. Message queues are a method of communication between applications. Messages can be returned immediately after being sent. There is a message system to ensure reliable and special delivery of information. In some implementations, the Kafka message queue is used. Kafka can process hundreds of thousands of messages per second with a delay as low as a few milliseconds. A single machine can support thousands of clients to read and write simultaneously.

[0034] In some embodiments, although warm data is not frequently queried every day, it will be used in specific business cycles or analysis needs, which requires both query efficiency and storage cost. In some embodiments, warm data is stored in a relational database or a non-relational database (NoSQL database), i.e., the second type of database. For example, a relational database uses a relational database management system (MySQL) partition table, and a non-relational database uses a database based on distributed file storage (MongoDB), which meets both query efficiency and storage cost.

[0035] In one implementation mode of the present application, warm data is synchronized and managed through an incremental synchronization strategy. Incremental synchronization refers to synchronizing only new data to the database after the data source is updated, thereby achieving real-time synchronization of data. Compared with full synchronization, incremental data synchronization can reduce network traffic for data transmission and improve synchronization speed and efficiency. As another implementation mode, warm data is synchronized through a scheduled task strategy. The scheduled task strategy refers to the scheduled synchronization of warm data to the database based on time periods as management nodes, thereby achieving scheduled synchronization of data.

[0036] The access to cold data in this application is usually sporadic, and the query speed requirement is not high, so it is stored in a low-cost, large-capacity cloud storage service (such as AWS S3 Glacier, Hadoop HDFS), that is, a third-class database. As another implementation method, it can also be stored in a low-cost tape library. The cold data of this application is synchronously managed on demand or in a periodic archiving manner.

[0037] The adaptation layer of the present application includes an input adapter and an output adapter. The input adapter is used to configure the input interface, and input the data generated according to the user behavior and the hot data, warm data and / or cold data retrieved from different databases into the business logic layer; the output adapter is used to configure the output interface, and transmit the data of the user information and / or account information in the business logic layer to different target terminals, and output the hot data, warm data and / or cold data obtained by hierarchical processing to different databases. The input adapter can use API Gateway, and use the input adapter as the only entrance for all external access to the business logic layer, providing authentication, authorization, current limiting, routing and other functions; the adapter connects to the database and message queue, etc.; the adaptation layer is mainly responsible for communicating with the outside world and isolating the technical details. The input adapter and the output adapter of the adaptation layer interact with the interface clearly defined by the interface interaction layer, and the interfaces are loosely coupled. When adding or replacing the database type, only the corresponding adapter needs to be adjusted, which is convenient for the replacement and expansion of the technology stack, and the business logic layer will not be affected.

[0038] The adaptation layer in this application is based on security protocols such as OAuth2 and JWT to ensure the security of data transmission and encrypt sensitive data, such as HTTPS protocol encrypted transmission; in addition, the interface of the interface interaction layer adopts standardized RESTfulAPI specifications, including request format, response format, error code, etc., to ensure consistency and ease of use across system calls.

[0039] Different interfaces of the interface interaction layer are used to connect to different target ends, including internal services and other channel applications. In this example, internal services refer to other services in the banking system that have data access and storage relationships with the business logic layer, and channel applications are other financial institutions. The Dubbo adapter is used between the business logic layer and the internal services, the Http adapter is used between the channel application and the business logic layer, the Repositry adapter is used between the adaptation layer and the repository, and the MO adapter is used for the Kafka message queue and its Rocketmq.

[0040] The data management system of this application integrates advanced authentication, authorization, current limiting, and routing mechanisms at the API gateway level. Through unified management strategies, banks can easily adjust security policies and performance parameters to ensure that the system can meet strict compliance requirements and efficiently handle the growing account data management needs. In addition, the highly modular API gateway architecture supports third-party developers to develop customized plug-ins for specific security or performance optimization needs, promotes the formation of a technological innovation ecosystem, and enables banks to quickly adapt to future technological developments and business changes.

[0041] like Figure 2As shown, when the application data management system responds to a storage request, it performs the following steps: S10, the business logic layer performs hierarchical processing on the user information and / or account information generated by the target end to obtain hot data, warm data and / or cold data.

[0042] S20, the adaptation layer receives hot data, warm data and / or cold data, and transmits the hot data, warm data and / or cold data in different ways, and transmits them to the corresponding first type database, second type database and / or third type database via the interface interaction layer.

[0043] In one embodiment, the business logic layer is used to perform hierarchical processing on user information and account information to obtain hot data, warm data and / or cold data; the adaptation layer receives the hot data, warm data and cold data output by the business logic layer, and transmits the hot data through a message queue, and then stores the hot data in a first type of database through an interface interaction layer connected to the adaptation layer, transmits the warm data through an incremental synchronization strategy, and then stores the warm data in a second type of database through an interface interaction layer connected to the adaptation layer, transmits the cold data through periodic archiving, and then stores the cold data in a third type of database through an interface interaction layer connected to the adaptation layer.

[0044] like Figure 3 As shown, when the application data management system responds to a read request, the following steps are performed: S30, the adaptation layer retrieves hot data from the first type of database, retrieves warm data from the second type of database and / or retrieves cold data from the third type of database through the interface interaction layer, and inputs the retrieved hot data, warm data and / or cold data into the business logic layer.

[0045] S40, the business logic layer identifies hot data, warm data and / or cold data through a user model and / or an account model and outputs corresponding user information and / or account information.

[0046] S50, the adaptation layer outputs the corresponding user information and / or account information to the corresponding target end through the interface interaction layer.

[0047] In one embodiment, the interface interaction layer is used to receive hot data retrieved from the first type of database, warm data retrieved from the second type of database, and cold data retrieved from the third type of database; the retrieved hot data, warm data, and / or cold data are input into the business logic layer, and the business logic layer processes the received hot data, warm data, and / or cold data, and reads out the received data through the account model and / or account model after processing; the read user information and / or account information is transmitted to different target ends connected to the interface interaction layer. Specifically, in response to a read request, hot data in the first type of database is read through the cache area, while warm data and cold data are directly read from the second type of database and the third type of database, respectively. When hot data is read, the business logic layer queries the cache area: when hot data is in the cache area, hot data is read from the cache area; when hot data is not in the cache area, hot data is automatically loaded from the first type of database, and the hot data in the cache area is updated.

[0048] This application uses optimistic locking or pessimistic locking for both caching and updating to prevent concurrency conflicts and ensure the correctness of data under high concurrency. During the update process, the business logic layer notifies the update instructions through the publish / subscribe model, where the update instructions include adding data or invalidating corresponding entries; then the adaptation layer uses optimistic locking or pessimistic locking strategies to transmit and process the hot data, warm data and / or cold data output by the business logic layer according to the update instructions, and updates the hot data, warm data and / or cold data to different databases through the interface interaction layer. During the caching process, the business logic layer notifies the cache update instructions through the publish / subscribe model; the adaptation layer uses optimistic locking or pessimistic locking strategies according to the cache instructions to retrieve hot data from the first type of database, and loads the retrieved hot data into the cache area for updating.

[0049] For high-concurrency scenarios, use tools in the Java concurrency package (such as ConcurrentHashMap and Semaphore) to ensure the security and consistency of data updates.

[0050] Among them, pessimistic locking means always assuming the worst case scenario. Every time you get data, you assume that others will modify it, so you lock it every time you get data. That is, shared resources are only used by one thread at a time, and other threads are blocked. After use, the resources are transferred to other threads. Exclusive locks such as synbchronized and ReentrantLock in Java are based on the idea of ​​pessimistic locking.

[0051] Among them, optimistic locking means always assuming the best situation. Every time you get data, you will assume that others will not modify it, so you will not lock it. However, when updating, you will determine whether others have updated the data in the next device. You can use the version mechanism and CAS algorithm to implement it. In Java, the atomic variable class under the java.util.concurrent.atomic package uses CAS, an implementation method of optimistic locking. Optimistic locking is suitable for multi-read application types and can improve throughput.

[0052] In some embodiments of the present application, the management system also includes an auxiliary management layer, which includes a transaction log and a version control unit. During the data flow and synchronization process, the transaction log and / or the version control unit are used for auxiliary control management to ensure the consistency of the bank's data during the synchronization strategy.

[0053] The data management system of the present application greatly shortens the query response time through data hierarchical storage and intelligent caching strategies, especially the memory management of hot data, and can achieve almost real-time data access, significantly improve the speed and satisfaction of customers' account information inquiries, and enhance user experience; data hierarchical storage also helps to jointly allocate resources and reduce unnecessary data redundancy. The business logic layer and its unified access interface simplify the integration complexity between systems and reduce subsequent maintenance and upgrade costs; the data management mechanism and the use of different synchronization strategies for different data ensure data consistency across systems, reduce errors and inconsistencies caused by data synchronization delays, and improve the reliability of decision-making and the accuracy of business processing; therefore, the highly decoupled data management system of the present application can enable banks to deploy new services or adjust existing services more quickly without worrying about excessive impact on existing systems, thereby enhancing the business flexibility and market competitiveness of banks and quickly responding to changing customer needs and market trends.

[0054] The following describes the bank's data management method in conjunction with some specific embodiments.

[0055] Figure 4 A flow chart of a data management method according to an embodiment of the present application is shown. Exemplarily, the data management method includes the following steps: S110, classifying the data into hot data, warm data and cold data according to the access frequency of the data.

[0056] This application generates bank data based on user behavior. In order to better manage the bank's data and improve the query efficiency during the management process, this application performs hierarchical processing on the bank's data. Specifically, the bank's data is divided into three levels according to the access frequency of the bank's data, namely hot data, warm data and cold data. In this example, hierarchical processing is performed based on the access frequency of the bank's data. As another implementation method, hierarchical processing is performed based on the access frequency and importance of the bank's data. The combination of access frequency and importance has a better effect on the hierarchical processing of bank data.

[0057] S120, in response to the storage request, the business logic layer performs hierarchical processing on the input data to obtain hot data, warm data and / or cold data.

[0058] S130, the adaptation layer receives the hot data, warm data and / or cold data output by the business logic layer, and transmits it to the interface interaction layer in different ways.

[0059] The adaptation layer receives hot data, warm data and / or cold data output by the business logic layer, and transmits the received hot data through a message queue, transmits the received warm data through an incremental synchronization strategy, and / or transmits the received cold data to the interface interaction layer through periodic archiving.

[0060] S140, the interface interaction layer stores the hot data in the first type of database, stores the warm data in the second type of database, and stores the cold data in the third type of database.

[0061] In this application, hot data is crucial to the user's instant experience and requires a fast response, so it is suitable for storage in an in-memory database and / or a high-performance cache to ensure millisecond query speeds; although warm data is not frequently queried every day, it will be used in specific business cycles or analysis needs, requiring both query efficiency and storage costs; access to cold data is usually sporadic, and the query speed requirement is not high, so it is stored in low-cost, large-capacity cloud storage services (such as AWS S3 Glacier, Hadoop HDFS), and as another implementation method, it can also be stored in a low-cost tape library.

[0062] In some implementations of the present application, a transaction log or version control system is used to assist in the data flow and synchronization process to ensure the consistency of the bank's data during the synchronization strategy process.

[0063] Figure 5 A flow chart of a data management method according to an embodiment of the present application is shown. Exemplarily, the data management method includes the following steps: S210, in response to the read request, the business logic layer accesses and retrieves the hot data, warm data and cold data via the adaptation layer.

[0064] In the present application, a read request corresponding to the bank's data is generated based on consumption; when hot data needs to be read, the hot data in the first category database is first loaded into the cache area, and then the hot data in the cache area is accessed and read; when warm data needs to be read, the second category database is directly accessed, and the warm data is read from the second category database; when cold data needs to be read, the third category database is directly accessed, and the cold data is read from the third category database.

[0065] S220, the business logic layer identifies the received hot data, warm data and / or cold data and outputs the corresponding user information and / or account information.

[0066] The business logic layer identifies hot data, warm data and / or cold data through user models and / or account models and outputs corresponding user information and / or account information.

[0067] S230, the adaptation layer outputs the corresponding user information and / or account information to the corresponding target end through the interface interaction layer.

[0068] In response to a read request, the present application transmits the read hot data, warm data and / or cold data to different target ends connected to the interface interaction layer via different transmission protocols of the adaptation layer.

[0069] It can be understood that the management method of this embodiment corresponds to the management system of the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described repeatedly here.

[0070] The present application also provides a terminal device. Exemplarily, the terminal device includes a processor and a memory, wherein the memory stores a computer program, and the processor runs the computer program to enable the terminal device to execute the functions of the various modules in the above-mentioned data management method or the above-mentioned management system.

[0071] Among them, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application.

[0072] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electric erasable programmable read-only memory (EEPROM), etc. The memory is used to store a computer program, and the processor may execute the computer program accordingly after receiving an execution instruction.

[0073] The present application also provides a computer-readable storage medium for storing the computer program used in the above terminal device. For example, the computer-readable storage medium may include but is not limited to: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0074] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0075] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0076] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0077] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data management system, characterized in that: include: Business logic layer, adaptation layer and interface interaction layer; the data includes user information and account information opened by the user; The business logic layer includes an account model and a user model, wherein the user model is used to identify the user information and the account model is used to identify the account information; the interface interaction layer is used to connect different storage devices and different target terminals; When responding to a storage request, the business logic layer performs hierarchical processing on the user information and / or the account information generated by the target end to obtain hot data, warm data and / or cold data; the adaptation layer transmits the obtained hot data through a message queue and transmits it to the first type of database via the interface interaction layer, and / or transmits the obtained warm data through an incremental synchronization strategy and transmits it to the second type of database via the interface interaction layer, and / or transmits the obtained cold data through periodic archiving and transmits it to the third type of database via the interface interaction layer.

2. The data management system according to claim 1, characterized in that: The data management system also includes: In response to a read request, the adaptation layer retrieves the hot data from the first type of database, retrieves the warm data from the second type of database, and / or retrieves the cold data from the third type of database through the interface interaction layer, and inputs the retrieved hot data, the warm data, and / or the cold data into the business logic layer; the business logic layer identifies the hot data, the warm data, and / or the cold data through the user model and / or the account model and outputs the corresponding user information and / or account information; the adaptation layer outputs the corresponding user information and / or account information to the corresponding target end through the interface interaction layer.

3. The data management system according to claim 2, characterized in that: The business logic layer identifies the hot data, which includes: The business logic layer queries the cache area: When the hot data is in the cache area, reading the hot data from the cache area; When the hot data is not in the cache area, the hot data is automatically loaded from the first type of database, and the hot data in the cache area is updated.

4. The data management system according to claim 3, characterized in that: The updating of the hot data in the cache includes: The business logic layer triggers cache instructions through a publish / subscribe mode; The adaptation layer uses an optimistic lock or a pessimistic lock strategy to retrieve the hot data from the first type of database according to the cache instruction, and loads the retrieved hot data into the cache area for updating.

5. The data management system according to claim 1, characterized in that: The adaptation layer includes an input adapter and an output adapter; The input adapter is used to configure the input interface and transmit the data generated according to the user behavior and the hot data, the warm data and / or the cold data retrieved from different databases to the business logic layer; The output adapter is used to configure the output interface, transmit the user information and / or the account information in the business logic layer to different target ends, and transmit the hot data, the warm data and / or the cold data obtained by hierarchical processing to different databases.

6. A data management method, characterized in that: Applicable to the data management system according to any one of claims 1 to 5; the data management method comprises: In response to the storage request, the business logic layer performs hierarchical processing on the input data to obtain hot data, warm data and / or cold data; The adaptation layer receives the hot data, the warm data and / or the cold data output by the business logic layer, and transmits the received hot data through a message queue, transmits the received warm data through an incremental synchronization strategy, and / or transmits the received cold data to the interface interaction layer through periodic archiving; The interface interaction layer stores the hot data in the first type of database, stores the warm data in the second type of database, and stores the cold data in the third type of database.

7. The data management method according to claim 6, characterized in that: The business logic layer processes the data in a graded manner according to the access frequency and importance level of the data to obtain the hot data, the warm data and / or the cold data respectively.

8. A data management method, characterized in that: Applicable to the data management system according to any one of claims 1 to 5; the data management method comprises: In response to a read request, when the hot data needs to be read, the adaptation layer first loads the hot data in the first type of database into a cache area, and then the business logic layer accesses and reads the hot data in the cache area; When the warm data needs to be read, the business logic layer accesses the second-type database via the adaptation layer, and reads the warm data from the second-type database; When the cold data needs to be read, the business logic layer accesses the third type of database via the adaptation layer, and reads the cold data from the third type of database; The business logic layer identifies the read hot data, the warm data and / or the cold data, and outputs corresponding user information and / or account information after identification; The adaptation layer outputs the corresponding user information and / or account information to the corresponding target end through the interface interaction layer.

9. A terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the data management method according to any one of claims 6 to 8.

10. A computer-readable storage medium, characterized in that: The computer program stores a computer program, which, when executed on a processor, implements the data management method according to any one of claims 6 to 8.

Citation Information

Patent Citations

  • Data management system and data management method based on cold data, temperature data and hot data

    CN110083649A

  • Data updating method and device, terminal equipment and storage medium

    CN116303511A

  • Application management method, computing device and storage medium

    CN117667124A

  • Data processing method, device and equipment

    CN118313926A

Cited By

  • Data sharing method, system, device, equipment, storage medium and program product

    CN121387186A