Data processing method and device, server and storage medium

Abstract

The application provides a data processing method and device, a server and a storage medium. The target storage data is obtained, the target storage data being data in a database or page maintenance data of a client. Then, the target storage data is processed based on a preset key-value pair construction rule to obtain a cache key-value pair corresponding to the target storage data. The key name of the cache key-value pair is used to indicate the data type of the target storage data, and the key value of the cache key-value pair is used to indicate the target storage field of the target storage data and the attribute value of the target storage field. Finally, the key-value pair is stored in the cache. In the technical solution, the server can construct a special key-value pair based on the characteristics of the data itself, so as to facilitate the query of the required data from the cache in a specific scenario. The hit rate of the query in the cache and the adaptability of the complex scene application are improved, resource waste is avoided, and the access efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, server and storage medium. Background Technology

[0002] As application systems continue to expand, the amount of data stored in databases is increasing. Compared with application systems in other fields, internet finance systems contain a wide variety of financial data types with complex business meanings. Given the business application scenarios they are in, ensuring the rapid and accurate acquisition of financial data that meets business needs in complex business scenarios has become more urgent and important.

[0003] In existing technologies, data storage and retrieval schemes involve storing the data that needs to be stored in a database, while storing a portion of the data as a whole object entity in a cache. When a business scenario requires a certain piece of data, it first queries the cache; if the data is not found in the cache, it then queries the database to obtain the target data.

[0004] However, in practical applications, not all fields in the data table will be used. Storing and retrieving the entire object entity data not only wastes cache resources, but also has a low hit rate and low access efficiency. Summary of the Invention

[0005] This application provides a data processing method, apparatus, server, and storage medium to overcome the problems of wasted cache resources, low hit rate, and low access efficiency in existing data access schemes.

[0006] Firstly, this application provides a data processing method, including:

[0007] Obtain target stored data, which is data in a database or page maintenance data on the client side;

[0008] Based on the preset key-value pair construction rules, the target stored data is processed to obtain the cached key-value pairs corresponding to the target stored data. The key name of the cached key-value pairs is used to indicate the data type of the target stored data, and the key value of the cached key-value pairs indicates the target stored field and the attribute value of the target stored field.

[0009] The key-value pairs are stored in the cache.

[0010] In one possible design of the first aspect, the target stored data is data in a database;

[0011] Accordingly, acquiring the target stored data includes:

[0012] Obtain a data storage request, wherein the data storage request includes: a business scenario identifier;

[0013] Based on the business scenario identifier, the hot data set corresponding to the business scenario identifier in the database is selected, and each hot data in the hot data set is data whose usage frequency is greater than a preset frequency threshold;

[0014] Based on the business attribute information of the data, the hot data in the hot data dataset is classified to obtain the target storage data for each category.

[0015] In another possible design of the first aspect, the target stored data is the client's page maintenance data;

[0016] Accordingly, the method further includes:

[0017] The target storage data is stored in the database.

[0018] In another possible design of the first aspect, the step of processing the target stored data based on preset key-value pair construction rules to obtain cached key-value pairs corresponding to the target stored data includes:

[0019] The data type of the target stored data is determined based on the business attribute information of the target stored data;

[0020] Based on the data type of the target stored data, determine the identifying field and at least one class distinguishing field of the target stored data;

[0021] Based on the identifying field and the at least one class distinguishing field, the type identifier of the target stored data is determined, and the type identifier serves as the key name of the key-value pair;

[0022] Determine the target storage field and the attribute value of the target storage field for the target stored data;

[0023] Determine the class attribute name of the target stored field in the mapper mapping file of the data access layer;

[0024] Construct the key-value pair of the cache based on the class attribute name and the attribute value of the target storage field.

[0025] In yet another possible design of the first aspect, the method further includes:

[0026] Obtain a data read request, the data read request including: at least one target key name;

[0027] Based on the at least one target key name, determine the target key-value pair corresponding to each target key name in the cache;

[0028] From all the identified target key-value pairs, obtain the key value in each target key-value pair, wherein the key value includes: at least one target field and the attribute value of each target field;

[0029] Based on the pipeline pattern of the cache, a data read response is fed back, which includes the key-value pairs of all target key-value pairs.

[0030] Optionally, before determining the target key-value pair corresponding to each target key name in the cache based on the at least one target key name, the method further includes:

[0031] Store the at least one target key name into a string array;

[0032] Accordingly, determining the target key-value pair corresponding to each target key name in the cache based on the at least one target key name includes:

[0033] Based on each target key name in the string array, the cache is queried sequentially to obtain the key value corresponding to each target key name.

[0034] Secondly, this application provides a data processing apparatus, comprising:

[0035] The acquisition module is used to acquire target stored data, which is data in a database or page maintenance data on a client.

[0036] The processing module is used to process the target stored data based on a preset key-value pair construction rule to obtain a cached key-value pair corresponding to the target stored data. The key name of the cached key-value pair is used to indicate the data type of the target stored data, and the key value of the cached key-value pair indicates the target stored field and the attribute value of the target stored field.

[0037] A storage module is used to store the key-value pairs in a cache.

[0038] In one possible design of the second aspect, the target stored data is data in a database;

[0039] Accordingly, the acquisition module is specifically used for:

[0040] Obtain a data storage request, wherein the data storage request includes: a business scenario identifier;

[0041] Based on the business scenario identifier, the hot data set corresponding to the business scenario identifier in the database is selected, and each hot data in the hot data set is data whose usage frequency is greater than a preset frequency threshold;

[0042] Based on the business attribute information of the data, the hot data in the hot data dataset is classified to obtain the target storage data for each category.

[0043] In another possible design of the second aspect, the target stored data is the client's page maintenance data;

[0044] Correspondingly, the storage module is also used to store the target storage data in the database.

[0045] In another possible design of the second aspect, the processing module is specifically used for:

[0046] The data type of the target stored data is determined based on the business attribute information of the target stored data;

[0047] Based on the data type of the target stored data, determine the identifying field and at least one class distinguishing field of the target stored data;

[0048] Based on the identifying field and the at least one class distinguishing field, the type identifier of the target stored data is determined, and the type identifier serves as the key name of the key-value pair;

[0049] Determine the target storage field and the attribute value of the target storage field for the target stored data;

[0050] Determine the class attribute name of the target stored field in the mapper mapping file of the data access layer;

[0051] Construct the key-value pair of the cache based on the class attribute name and the attribute value of the target storage field.

[0052] In another possible design of the second aspect, the acquisition module is further configured to acquire a data read request, the data read request including: at least one target key name;

[0053] The processing module is further configured to:

[0054] Based on the at least one target key name, determine the target key-value pair corresponding to each target key name in the cache;

[0055] From all the identified target key-value pairs, obtain the key value in each target key-value pair, wherein the key value includes: at least one target field and the attribute value of each target field;

[0056] Based on the pipeline pattern of the cache, a data read response is fed back, which includes the key-value pairs of all target key-value pairs.

[0057] Optionally, the storage module is further configured to store the at least one target key name into a string array;

[0058] Correspondingly, the processing module is also used to query the cache sequentially based on each target key name in the string array to obtain the key value corresponding to each target key name.

[0059] Thirdly, this application provides a server, including: a memory and a processor;

[0060] The memory is used to store computer-executable instructions that can run on the processor;

[0061] When the processor executes the computer execution instructions, it implements the method as described in the first aspect and various possible designs above.

[0062] Alternatively, the processor described above can be a chip.

[0063] Fourthly, this application provides a computer-readable storage medium storing computer instructions that, when executed by a processor, are used to implement the methods described in the first aspect and various possible designs above.

[0064] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the first aspect and various possible designs.

[0065] The data processing method, apparatus, server, and storage medium provided in this application embodiment acquire target stored data, which may be data in a database or page maintenance data of a client. Then, based on preset key-value pair construction rules, the target stored data is processed to obtain cached key-value pairs corresponding to the target stored data. The key name of the cached key-value pair indicates the data type of the target stored data, and the key value indicates the target stored field and its attribute value. Finally, the key-value pair is stored in the cache. In this technical solution, the server can construct dedicated key-value pairs based on the characteristics of the data itself, facilitating the retrieval of data that meets requirements from the cache in specific scenarios. This improves the hit rate of cache queries and the adaptability to complex application scenarios, not only avoiding resource waste but also improving access efficiency. Attached Figure Description

[0066] Figure 1 A schematic diagram illustrating the application scenarios to which the data processing method provided in this application is applicable;

[0067] Figure 2 A flowchart illustrating an embodiment of the data processing method provided in this application;

[0068] Figure 3A flowchart illustrating Embodiment 2 of the data processing method provided in this application;

[0069] Figure 4 A flowchart illustrating Embodiment 3 of the data processing method provided in this application;

[0070] Figure 5 A schematic diagram of the structure of an embodiment of the data processing method provided in this application;

[0071] Figure 6 This is a schematic diagram of the structure of the server embodiment provided in this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] Before introducing the application scenarios and technical solutions of this application, let's first introduce some of the terms involved in this application:

[0074] The SSM framework is an integration of Spring MVC, Spring, and MyBatis frameworks. It follows the standard MVC pattern, dividing the entire system into four layers: presentation layer, controller layer, service layer, and DAO layer. Spring MVC handles request forwarding and view management, Spring manages business objects, and MyBatis serves as the data object persistence engine. Specifically, the Service layer is the business layer, focusing on business logic. Database operations are handled by the DAO layer. The DAO layer encapsulates database access: CRUD operations (Create, Read, Update, Delete), without involving business logic; it simply retrieves specific data based on certain conditions.

[0075] Redis: Completely open source, compliant with the Berkeley Software Distribution (BSD) license, is a high-performance key-value database.

[0076] Hash data structure: Essentially, it's a data structure that extends an array. A hash table can be described as an optimization of an array that supports direct indexing of data (values), aiming to perform operations such as adding, deleting, modifying, and searching data in constant time.

[0077] Key: Used to locate the index in the query cache; in this embodiment, it is referred to as the key name of the key-value pair.

[0078] Value: The data stored in the cache, referred to as the key value of the key-value pair in this embodiment of the application;

[0079] The Mapper.xml file defines the Structured Query Language (SQL) for database operations. Each SQL statement is a Statement, a crucial interface in Java for performing database operations. It's used to send SQL statements to the database after an established connection has been established. Statement objects are used to execute simple SQL statements without parameters. This Mapper file is the core of the MyBatis framework.

[0080] For example, the specific meaning of the Mapper mapping file is explained through the following procedure:

[0081] <? xml version="1.0" encoding="UTF-8"? >

[0082] <! DOCTYPE mapper

[0083] PUBLIC"- / / mybatis.org / / DTD Mapper 3.0 / / EN"

[0084] "http: / / mybatis.org / / dtd / mybatis-3-mapper.dtd"> / / Mybatis DTD constraints

[0085] <mapper namespace="test"> / / mapper is the root tag, namespace="test" is the namespace, and together with the ID attribute, they form a unique identifier for a SQL statement.

[0086] <select id＝＂selectuserlist＂resulttype＝＂com,qzw.bean.user＂> / / select is a query operation, there are also insert, update, delete; id="selectUserList" is the ID of the SQL statement, which, together with the namespace, forms a unique identifier for a SQL statement; resultType="com,qzw.bean.User" indicates the data type of the result

[0087] select * from user / / SQL statement

[0088] < / select>

[0089]

[0090] The application scenarios of this application are explained below:

[0091] With the development of the internet, large-scale web-based application systems based on browsers have become increasingly common. This is accompanied by increasingly complex business application scenarios and a continuous expansion in system access volume and data generation, especially in the financial industry. Large commercial banks typically have tens of millions of customers and accounts, and financial data types are complex. In specific business transaction scenarios, there's a need to frequently retrieve specific types of financial data that meet certain conditions, such as querying specific price data from a bond valuation record within a specified data source, data type, and trading venue. Directly querying the database in such cases is inefficient, especially when query requests surge, significantly slowing response times and consuming excessive system resources. However, storing the entire data record in a Redis cache wastes cache space, storing redundant data and failing to meet the data demands of complex scenarios.

[0092] Therefore, in practical applications, it is necessary to construct a suitable key based on the characteristics of financial data and the application scenario, and at the same time, select an appropriate data structure to store the required data. Thus, how to choose a suitable data structure to quickly improve the system's response speed in business scenarios and the accuracy of data in the query cache becomes particularly important and an urgent problem to be solved.

[0093] As described in the background section, current data storage solutions store data in a database while also storing some data in a cache. For example, different data tables of different data types are created in the database, and an entire record from a frequently used table is stored in the cache. Correspondingly, when a business scenario requires certain data, the server can first query the cache based on the client's query request. If the data is not found in the cache, it will then query the database to obtain the response data. Specifically, the corresponding stored data is first queried one by one from the cache or the database, and then the required fields are selected.

[0094] The above analysis shows that the existing technology has two main drawbacks:

[0095] 1. The query process is time-consuming and contains redundant data, wasting cache resources.

[0096] The common practice is to serialize a single entity object from a frequently used table in the database into a JSON object representation and then store it in Redis. When relevant data is needed, the JSON is first retrieved from the cache, then deserialized to obtain the entire entity data, and finally the required fields are extracted from that entity data record. This approach has the drawback of being time-consuming for serializing entity data. Furthermore, since not all fields of a single data record are used in practice, redundant data is stored in the cache, wasting cache resources. Therefore, choosing an appropriate data structure to store the data (e.g., financial data) is crucial.

[0097] 2. Low hit rate and poor performance.

[0098] Because financial data is complex and lacks categorization, storing it uniformly in the cache is inconvenient for querying and reduces the Redis hit rate. In practical applications, financial data often involves numerous technical terms. The naming of cache keys needs to be standardized and have clear meanings; otherwise, it will be very inconvenient for various modules in the application system to query data. Therefore, in complex business scenarios, the naming of cache keys is crucial. If the key name cannot accurately describe the characteristics of the entire record, it will be difficult to accurately retrieve financial data that meets the query requirements, and it will also be difficult to fully utilize the high efficiency of cache read and write performance.

[0099] In summary, enterprise-level web application systems based on the SSM framework often experience very frequent data interactions between the front-end and back-end in certain specific application scenarios. Under high concurrency, a large number of front-end read and write requests flood the database, putting immense pressure on it and slowing down the overall system response. As the application system expands, the amount of data stored in the database grows significantly. Compared to application systems in other fields, internet finance systems contain a wide variety of financial data types with complex business meanings, and their application scenarios are complex. Therefore, ensuring the rapid and accurate acquisition of financial data that meets business needs in complex business scenarios becomes increasingly urgent and important.

[0100] In the embodiments of this application, a data processing method is provided, specifically a cache data access and synchronization processing method based on Redis storage data structure. According to business scenario requirements, financial hot data from existing database tables is filtered out. These hot data are then classified according to different types, and key-value pairs are constructed based on preset key-value pair construction rules. Specifically, according to the characteristics of the data itself, fields with obvious identification significance and distinguishability in the data record in the database table are selected, and exclusive type labels are constructed, which serve as the keys when storing them in Redis. Simultaneously, fields that need to be frequently used in the business scenario are determined, and a hash data structure in Redis is selected as the storage object. The class attribute name of this field in the persistent mapping layer (Dao layer) mapper mapping file is used as the field ID, and its corresponding field value is used as the value. The entire hash object is stored in the cache. During the processing, the business logic layer (service layer) simultaneously flushes the data into the cache, saving Redis cache storage space, reducing performance loss, optimizing cache resource utilization, and improving storage and retrieval efficiency.

[0101] Optionally, the key points of the technical solution of this application will be summarized below:

[0102] 1. Based on the requirements of the business scenario, the target stored data (financial hot data) was first clearly classified. Then, combined with the characteristics of the target stored data itself, a unique key with obvious identification meaning was constructed according to the preset key-value pair construction rules, which can improve the accuracy of data retrieval in the cache.

[0103] 2. The hash data structure is highly targeted and applicable to data in business scenarios (e.g., bond valuation data, bond rating data, exchange rate information, and interest rate information in financial business scenarios). It only stores the field information that needs to be used, removes redundant data, and saves storage space for Redis cache.

[0104] For example, Figure 1 A schematic diagram illustrating the application scenario to which the data processing method provided in this application is applicable. (Refer to...) Figure 1 As shown in the diagram, the application scenario may include: multiple client devices (exemplary, client device 111 and client device 112), a server 12 connected to each client device, a database 13 and a cache 14 connected to the server 12, and the database 13 and the cache 14 may also be directly connected.

[0105] In practical applications, customer service personnel can interact with server 12 through client devices based on user needs or by receiving requests from terminal devices, generating various types of data. For example, in financial scenarios, the generated data may include, but is not limited to, bond valuation data, bond rating data, exchange rate information, interest rate information, etc. For subsequent queries, server 12 needs to store the generated data in database 13 and store some data in cache 14.

[0106] It is understood that server 12 can directly store the generated data in cache 14, or it can first store the generated data in database 13, and then store some data in the database in cache 14, or it can store the generated data in both database 13 and cache 14 at the same time. The specific method can be determined according to the actual scenario, and this embodiment does not limit it.

[0107] Optionally, in the embodiments of this application, the server 12 can execute the technical solution of this application, that is, after the server 12 obtains the target storage data, it can process the target storage data based on the preset key-value pair construction rules to obtain the cache key-value pairs of the target storage data and store them in the cache.

[0108] Optionally, in embodiments of this application, the application scenario diagram may further include terminal devices used by the user, for example, terminal device 113 and terminal device 114. Terminal device 113 and terminal device 114 can communicate with a client device (e.g., client device 111) to achieve the purpose of business processing.

[0109] Understandable, Figure 1 The illustrated scenario is merely an example. In practical applications, this scenario may also include other devices, such as computer equipment, and can be adjusted according to actual needs. This application does not limit the scope of the illustration.

[0110] In the above Figure 1 In the scenario diagram shown, the embodiments of this application do not limit the specific form of each device. For example, the client device and the terminal device can be a mobile phone, a tablet computer, a PC terminal, etc., which will not be listed here.

[0111] It is understood that in practical applications, the server can execute the technical solution of this application and achieve the purpose of data storage and retrieval, and the embodiments of this application do not limit it.

[0112] Below, in conjunction with the above Figure 1The illustrated scenario provides a detailed explanation of the technical solution of the present invention through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0113] Figure 2 This is a flowchart illustrating an embodiment of the data processing method provided in this application. Figure 2 As shown, the data processing method may include the following steps:

[0114] S201. Obtain the target stored data, which is data in the database or page maintenance data on the client.

[0115] In one possible design of this application embodiment, if data is updated in the database, and the updated data is frequently used data, then for the convenience of subsequent queries, it is necessary to synchronously store the updated data in the cache. It can be understood that in this possible design, the data that needs to be updated to the cache is referred to as the target stored data.

[0116] Therefore, in this possible design, the target stored data is data from a database.

[0117] Accordingly, obtaining the target stored data in S201 can be achieved through the following steps:

[0118] A1. Obtain a data storage request, which includes: a business scenario identifier;

[0119] Understandably, in this possible design, the server's data storage scheme can be based on pre-defined periodic automatic triggering, external storage commands, or other methods; this embodiment does not limit it. Therefore, when data in the database needs to be updated to the cache, a data storage request is first obtained. This data storage request can be automatically triggered by the database or obtained from an external source, which will not be elaborated here.

[0120] Since subsequent data read requests are usually made in specific business transaction scenarios, the data storage requests obtained above need to include a business scenario identifier in order to simplify data maintenance costs.

[0121] A2. Based on the business scenario identifier, filter out the hot data sets corresponding to the business scenario identifier in the database.

[0122] Each piece of hot data in this hot data set is data whose usage frequency is greater than a preset frequency threshold.

[0123] In this embodiment, the server parses the obtained data storage request to obtain the business scenario identifier. Then, based on the business scenario identifier, i.e. the usage requirements of the business scenario, the server can filter out the hot data set in the database. The hot data set is a collection of multiple hot data.

[0124] It is understandable that each piece of hot data in the hot data set is data whose usage frequency is greater than a preset frequency threshold. In other words, hot data is data that is frequently queried but not frequently modified or deleted.

[0125] A3. Based on the business attribute information of the data, classify the hot data in the hot data set to obtain the target storage data for each category.

[0126] Optionally, for hot datasets selected from the database based on business scenario identifiers, they can be classified according to different types. For example, they can be divided based on the business attribute information of the data to obtain the target storage data for each category.

[0127] For example, data in commercial banks can be divided into four categories according to business attributes: transaction data, customer data, credit data, and asset data. Each of these categories can be further subdivided into multiple categories based on its own characteristics, which will not be elaborated here.

[0128] In another possible design of this application embodiment, the target stored data may be data records maintained by the front-end page of the web application system, referred to as the client's page maintenance data.

[0129] For example, when maintaining a data record on the front-end page of a web application system based on the SSM framework, the server (specifically the business logic layer) can first determine the data type based on the business attributes of the data record during the processing, thereby obtaining the page maintenance data for the client.

[0130] S202. Based on the preset key-value pair construction rules, process the target stored data to obtain the cache key-value pairs corresponding to the target stored data.

[0131] In this context, the key name of the cached key-value pair is used to indicate the data type of the target stored data, and the key value of the cached key-value pair indicates the target stored field and the attribute value of that target stored field.

[0132] In practical applications, Redis (caching) is an open-source, network-enabled, in-memory, optional persistent key-value store database. Therefore, when storing target data in the cache, the first step is to construct key-value pairs.

[0133] Optionally, the server may store preset key-value pair construction rules. In this embodiment, the preset key-value pair construction rules are used to indicate the construction method of key names and key values. Therefore, after obtaining the target stored data, the server can process the target stored data based on the preset key-value pair construction rules to construct the key names and key values ​​corresponding to the target stored data respectively.

[0134] Specifically, based on preset key-value pair construction rules, the key corresponding to the target storage data can be constructed. Then, the target storage fields and their attribute values ​​that need to be stored can be selected from the target storage data and used as the key's value.

[0135] For details on the implementation of key-value pairs, please refer to the following: Figure 3 The descriptions in the illustrated embodiments will not be repeated here.

[0136] S203. Store the key-value pair in the cache.

[0137] In this step, once the server determines the key-value pair, it can store the key-value pair in the cache.

[0138] Understandably, when the target stored data is client-side page maintenance data, in order to ensure consistency between the data in the database and the cache, the server also needs to store the target stored data in the database. That is, while the target stored data is stored in the database, the corresponding key-value pairs can be stored in the cache.

[0139] The data processing method provided in this application involves acquiring target stored data, which may be data in a database or page maintenance data on a client side. Then, based on preset key-value pair construction rules, the target stored data is processed to obtain cached key-value pairs corresponding to the target stored data. The key name of the cached key-value pair indicates the data type of the target stored data, and the key value indicates the target stored field and its attribute value. Finally, the key-value pair is stored in the cache. In this technical solution, the server can construct dedicated key-value pairs based on the characteristics of the data itself, facilitating the retrieval of data that meets requirements from the cache in specific scenarios. This improves the hit rate of cache queries and the adaptability to complex application scenarios, avoiding resource waste and improving access efficiency.

[0140] For example, based on the above embodiments, Figure 3 This is a flowchart illustrating Embodiment Two of the data processing method provided in this application. Figure 3 As shown, the above S202 can be achieved through the following steps:

[0141] S301. Determine the data type of the target stored data based on the business attribute information of the target stored data.

[0142] Optionally, in order to improve the applicability of complex application scenarios, in the embodiments of this application, data storage is performed based on the business scenario. Therefore, when the target stored data is obtained, the business attribute information of the target stored data can be determined first, and then the data type of the target stored data can be determined based on the business attribute information. Then, the key name of the cache key-value pair can be constructed according to the data type of the target stored data and the target stored data.

[0143] S302. Based on the data type of the target stored data, determine the identifying field and at least one class distinguishing field of the target stored data.

[0144] Optionally, the target stored data exists in the form of a data table. Therefore, the server can select an identifier field A and at least one field B that distinguishes different types of data from the data table corresponding to the target stored data. Specifically, the identifier field and at least one class distinguishing field of the target stored data are used.

[0145] S303. Based on the identifying field and the at least one class distinguishing field, determine the type identifier of the target stored data, which serves as the key name of the aforementioned key-value pair.

[0146] In this embodiment, in order to construct a key-value pair specific to the target stored data, a defined identifier field and at least one class distinguishing field can be combined to construct a type identifier, which is used as the key (key name) specific to the target stored data.

[0147] For example, when the identifier field of the target stored data is A and at least one class distinguishing field is B, a type identifier A_B is constructed, and the type identifier A_B is used as the key of the target stored data.

[0148] Understandably, since there may be multiple discriminative fields in the target stored data, the type identifier must include all discriminative fields.

[0149] S304. Determine the target storage field of the target stored data and the attribute value of the target storage field.

[0150] Optionally, in this embodiment, while determining the identifying field and at least one class distinguishing field of the target stored data based on the data type of the target stored data, the server can also determine the fields in the target stored data that need to be used frequently. In this embodiment, these are referred to as target stored fields. Therefore, the server can construct cached key-value pairs based on the target stored fields and their attribute values.

[0151] S305. Determine the class attribute name of the target storage field in the mapper mapping file of the data access layer.

[0152] In this embodiment, a hash data structure in Redis can be selected as the storage object, that is, the key value of the cache key-value pair. In this case, when storing the target storage data into the cache, the class attribute name of the target storage field to be stored in the cache (Redis) in the mapper mapping file of the Dao layer (data access layer) is determined, and it is used as the field ID, and its corresponding field value is used as the value.

[0153] S306. Construct the key-value pair of the cache key-value pair based on the attribute name of the class and the attribute value of the target storage field.

[0154] Optionally, after determining the class attribute name and the attribute value of the target storage field, the key-value pairs for caching can be constructed. Specifically, the storage structure of the key-value (storage object) is {"field": the attribute name of the target storage field in the mapper mapping file, "value": the attribute value corresponding to the target storage field}.

[0155] Based on the key name determined in S303 and the key value determined in S306, key-value pairs corresponding to the target stored data can be generated. Specifically, the key-value pairs are represented in Redis as Key:Map: {Constructed exclusive Key: {"field": the attribute name of the target stored field in the mapping file, "value": the attribute value corresponding to the target stored field}}.

[0156] As can be seen from the above analysis, the technical solution of this application differs from the prior art in that the naming rules for key names in key-value pairs are different, which will be explained below.

[0157] For example, in existing storage formats, the key name in a key-value pair is basically not distinctive, but only roughly categorized. For example, {INFRA:MD}:marketdatadefinition:main, where the value in the key-value pair is the JSON serialized from the Entity instance object.

[0158] In the technical solution of this application, the Key in the key-value pair has a clear identifying meaning for the type and characteristics of the target stored data. The Key is constructed according to the preset key-value pair construction rules in the embodiments of this application, for example, {INFRA:MD}:currency:valuation010011_IB_CBOND_1, and the value in the key-value pair is the attribute value of the target stored field.

[0159] The data processing method provided in this application determines the data type of the target stored data based on its business attribute information, and then determines the identifier field and at least one class distinguishing field of the target stored data, thereby constructing a type identifier for the target stored data, i.e., the key name of the key-value pair. Further, it determines the target storage field and the attribute value of the target storage field, and constructs the key value of the cached key-value pair based on the class attribute name and attribute value of the target storage field in the mapper mapping file of the data access layer. In this technical solution, by constructing the key name and key value of the key-value pair based on the attribute information of the target stored data itself, and because the key name is unique and has clear identifying meaning, its query accuracy in the cache is improved.

[0160] For example, based on the above embodiments, Figure 4 This is a flowchart illustrating Embodiment 3 of the data processing method provided in this application. Figure 4 As shown, in this embodiment, the data processing method may further include the following steps:

[0161] S401. Obtain a data read request, which includes at least one target key name.

[0162] In the embodiments of this application, when a client needs to query multiple target data, it can send a data read request to the server and carry at least one target key name corresponding to the multiple target data in the data read request so as to provide index information for subsequent data queries.

[0163] S402. Based on at least one of the target key names mentioned above, determine the target key-value pair corresponding to each target key name in the cache.

[0164] Optionally, in this embodiment, the server can query the cache based on each target key name in the data read request to locate the target key-value pair corresponding to each target key name.

[0165] S403. From all the identified target key-value pairs, obtain the key value of each target key-value pair.

[0166] The key value includes at least one target field and the attribute value of each target field.

[0167] Optionally, for all target key-value pairs identified in the cache, the corresponding key value can be determined from each target key-value pair, thereby determining at least one target field corresponding to each target key name and the attribute value of each target field.

[0168] S404, a cache-based pipeline mode, provides a data read response that includes the key-value pairs from all target key-value pairs.

[0169] In the embodiments of this application, when the server does not respond, it can continue to receive data read requests sent by the client. However, for each data read request, the response data corresponding to each data read request will be fed back all at once after all the key values ​​corresponding to the target key names are determined. This attribute is the pipeline mode of Redis, which significantly improves the performance of the Redis service.

[0170] For example, in an embodiment of this application, prior to S402 described above, the server may further store at least one target key name in the data read request into a string array. Accordingly, S402 can be implemented through the following steps:

[0171] For each target key name in the string array, query the cache sequentially to retrieve the key value corresponding to each target key name.

[0172] In this embodiment, for example, when querying a certain type of financial data, such as when the data read request includes a target key name, i.e. when querying a single piece of data, Redis uses a transaction approach to execute multiple commands at once and in sequence. During the transaction execution process, the server can execute other client instructions without interrupting the transaction.

[0173] For example, when a data read request includes multiple target keys, i.e., when querying multiple data records, multiple target keys can be stored in a string array. Redis's pipeline mode can then be used to batch query the corresponding values ​​in the cache. This means that the server can still receive new request instructions before the client device has read the response to the old request. For each request, the server can provide a response result all at once, i.e., provide the values ​​corresponding to all keys all at once. This reduces the time cost of data round-trip transmission and improves the efficiency of Redis server's executable operations.

[0174] The data processing method provided in this application involves obtaining a data read request, which includes at least one target key name. Based on the at least one target key name, the method determines the target key-value pairs corresponding to each target key name in the cache. Then, it retrieves the key-value pairs from all determined target key-value pairs. Finally, based on a pipelined cache model, it sends back a data read response, which includes the key-value pairs from all target key-value pairs. In this technical solution, the server can use Redis's transactional approach to execute multiple commands sequentially and return the values ​​corresponding to all keys at once, reducing the time cost of round-trip data transmission and improving the operational efficiency of the Redis server.

[0175] The technical solution of this application will be explained and illustrated below through two specific embodiments in a financial scenario.

[0176] It is understood that, in the embodiments of this application, the purpose of the following two embodiments is to improve the access efficiency of financial data in the cache and to some extent solve the problem of data consistency between the cache and the database. The specific implementation scheme is as follows: Specific Implementation Example 1:

[0178] 1. Taking bond valuation in asset data as an example, a bond can be further divided into local currency bonds and foreign currency bonds. The bond code is an essential field for every bond, denoted as A, and has a clear identifying meaning. However, for local currency bonds, it is not possible to lock a unique local currency bond valuation information solely through the bond code. But the bond code and the place of circulation can determine a unique local currency bond valuation information. Therefore, the place of circulation field has the characteristic of distinguishing different types of data, denoted as B. The data source field can also distinguish different bond valuation data, denoted as C. Construct a type identifier A_B_C, which is used as the key for storing this bond in the cache.

[0179] 2. Based on actual financial business scenarios, the fields most frequently used in the valuation data of a local currency bond are intraday full price, net price, adjusted duration, and valuation basis point value; the fields most frequently used in the valuation data of a foreign currency bond are yield to maturity and yield to exercise.

[0180] 3. Select the hash data structure in Redis as the storage object, and finally generate a key-value pair Key:Map in Redis as {the constructed exclusive key: {"field": the attribute name of this field in the mapping file, "value": the attribute value corresponding to this field}}, and store the entire hash object (Key:Map) in the cache.

[0181] 4. When a complete record of bond valuation data is entered on the front-end page of the application system, a key is constructed according to the key naming rules. The fields that need to be stored in the cache are selected and refreshed into the cache at the same time as the data is entered into the database. At the same time, if a piece of data is deleted or modified through the front-end page, the cache will also be synchronized.

[0182] Practical verification has shown that the caching access method provided in this application effectively improves the query efficiency and hit rate of hot data in the cache, optimizes the use of cache resources, effectively removes redundant data, and maintains the consistency between the database and the cache to a certain extent. Specific Implementation Example 2:

[0184] 1. Taking exchange rate data in asset data as an example, the exchange rate information of a currency pair is divided into three types: spot, swap, and forward. The base currency and non-base currency of the currency pair are essential fields for each exchange rate information, denoted as A, which has obvious identification significance. The quotation type field is divided into spot, swap, and forward, so it can be determined that the quotation type field has the characteristic of distinguishing different types of data, denoted as B. Thus, the type identifier A_B is constructed, which is used as the key for storing this bond in the cache.

[0185] 2. Based on actual financial business scenarios, the most frequently used fields in a spot exchange rate data set are the spot exchange rate buying price, selling price, and mid-rate; the most frequently used fields in a forward exchange rate data set are the forward exchange rate mid-rate and exchange rate term type.

[0186] 3. Select the hash data structure in Redis as the storage object, and finally generate a key-value pair Key:Map in Redis as {the constructed exclusive key: {"field": the attribute name of this field in the mapping file, "value": the attribute value corresponding to this field}}, and store the entire hash object (Key:Map) in the cache.

[0187] 4. When a complete exchange rate data record is entered into the front-end page of the application system, a key is constructed according to the key naming rules. The fields that need to be stored in the cache are selected and refreshed into the cache at the same time as the data is entered into the database. At the same time, if a piece of data is deleted or modified through the front-end page, the cache will also be synchronized.

[0188] As can be seen from the above embodiments, the technical solution of this application has the following advantages: it eliminates Java's serialization and deserialization operations, improves storage and query efficiency, saves Redis cache storage space, stores only the information that is needed, removes redundant data, and is easy to maintain the stored data. It only requires operations on the key-value pairs in the Map, which simplifies the subsequent maintenance process. According to the business scenario, hot data is classified, and a dedicated key is constructed based on the characteristics of the data itself, which makes it easy to query data that meets the requirements from the cache in specific scenarios, improves the cache hit rate and adaptability to complex application scenarios.

[0189] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0190] Figure 5 This is a schematic diagram illustrating the structure of an embodiment of the data processing method provided in this application. Figure 5 As shown, the data processing apparatus may include:

[0191] The acquisition module 501 is used to acquire target stored data, which is data in a database or page maintenance data of a client.

[0192] The processing module 502 is used to process the target stored data based on a preset key-value pair construction rule to obtain a cached key-value pair corresponding to the target stored data. The key name of the cached key-value pair is used to indicate the data type of the target stored data, and the key value of the cached key-value pair indicates the target stored field and the attribute value of the target stored field.

[0193] Storage module 503 is used to store the key-value pairs in a cache.

[0194] As an example, the target stored data is data in a database;

[0195] Accordingly, the acquisition module 501 is specifically used for:

[0196] Obtain a data storage request, wherein the data storage request includes: a business scenario identifier;

[0197] Based on the business scenario identifier, the hot data set corresponding to the business scenario identifier in the database is selected, and each hot data in the hot data set is data whose usage frequency is greater than a preset frequency threshold;

[0198] Based on the business attribute information of the data, the hot data in the hot data dataset is classified to obtain the target storage data for each category.

[0199] As another example, the target stored data is the client's page maintenance data;

[0200] Correspondingly, the storage module 503 is also used to store the target storage data into the database.

[0201] In one possible design of this application embodiment, the processing module 502 is specifically used for:

[0202] The data type of the target stored data is determined based on the business attribute information of the target stored data;

[0203] Based on the data type of the target stored data, determine the identifying field and at least one class distinguishing field of the target stored data;

[0204] Based on the identifying field and the at least one class distinguishing field, the type identifier of the target stored data is determined, and the type identifier serves as the key name of the key-value pair;

[0205] Determine the target storage field and the attribute value of the target storage field for the target stored data;

[0206] Determine the class attribute name of the target stored field in the mapper mapping file of the data access layer;

[0207] Construct the key-value pair of the cache based on the class attribute name and the attribute value of the target storage field.

[0208] In another possible design of this application embodiment, the acquisition module 501 is further configured to acquire a data read request, the data read request including: at least one target key name;

[0209] The processing module 502 is further configured to:

[0210] Based on the at least one target key name, determine the target key-value pair corresponding to each target key name in the cache;

[0211] From all the identified target key-value pairs, obtain the key value in each target key-value pair, wherein the key value includes: at least one target field and the attribute value of each target field;

[0212] Based on the pipeline pattern of the cache, a data read response is fed back, which includes the key-value pairs of all target key-value pairs.

[0213] Optionally, the storage module 503 is further configured to store the at least one target key name into a string array;

[0214] Accordingly, the processing module 502 is also used to query the cache sequentially based on each target key name in the string array to obtain the key value corresponding to each target key name.

[0215] The apparatus provided in this application embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be repeated here.

[0216] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. These modules can be implemented entirely in software through processing element calls; they can also be implemented entirely in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. During implementation, each step of the above method or each of the above modules can be completed through the integrated logic circuits in the hardware of the processor element or through software instructions.

[0217] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to implement a system-on-a-chip (SOC).

[0218] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0219] Figure 6 This is a schematic diagram of the structure of the server embodiment provided in this application. Figure 6 As shown, the server may include a processor 601 and a memory 602.

[0220] The memory 602 is used to store computer programs, and the processor 601 is used to execute the computer programs to implement the technical solutions of the above method embodiments.

[0221] Optionally, the server may also include: a communication interface 603 and a system bus 604. The memory 602 and the communication interface 603 are connected to the processor 601 through the system bus 604 and complete communication between them. The communication interface 603 is used to communicate with other devices.

[0222] Optionally, the processor 601 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0223] The memory 602 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device.

[0224] The communication interface 603 can be implemented using a transceiver. This interface is used to enable communication between the database access device and other devices (such as clients, read-write databases, and read-only databases).

[0225] The system bus 604 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0226] Optionally, embodiments of this application also provide a computer-readable storage medium storing computer instructions, which, when executed by a processor, are used to implement the technical solutions of the above-described method embodiments.

[0227] Optionally, this application embodiment also provides a chip for executing instructions, the chip being used to execute the technical solutions of the above method embodiments.

[0228] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solutions of the above-described method embodiments.

[0229] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the preceding and following related objects; in formulas, the character " / " indicates a "division" relationship between the preceding and following related objects. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items.

[0230] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. In the embodiments of this application, the order of the above-mentioned process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized by, The method comprises the following steps: acquiring target storage data, the target storage data being data in a database or page maintenance data of a client; processing the target storage data based on a preset key-value pair construction rule to obtain a cache key-value pair corresponding to the target storage data, a key name of the cache key-value pair being used to indicate a data type of the target storage data, and a key value of the cache key-value pair being a target storage field of the target storage data and an attribute value of the target storage field; storing the key-value pair in a cache; the processing of the target storage data based on the preset key-value pair construction rule to obtain the cache key-value pair corresponding to the target storage data comprises the following steps: determining a data type of the target storage data according to business attribute information of the target storage data; determining an identification field and at least one class distinguishing field of the target storage data according to the data type of the target storage data; determining a type identifier of the target storage data according to the identification field and the at least one class distinguishing field, the type identifier being used as the key name of the key-value pair; determining a target storage field frequently used in a financial business of the target storage data and an attribute value of the target storage field; determining a class attribute name of the target storage field in a mapper mapping file of a data access layer; constructing the key value of the cache key-value pair according to the class attribute name and the attribute value of the target storage field.

2. The method of claim 1, wherein, The target storage data is data in a database. Correspondingly, the acquiring of the target storage data comprises the following steps: acquiring a data storage request, the data storage request comprising a business scenario identifier; filtering out a hot data set corresponding to the business scenario identifier in the database according to the business scenario identifier, each piece of hot data in the hot data set being data with a frequency of use greater than a preset frequency threshold; classifying hot data in the hot data set based on business attribute information of the data to obtain target storage data of each class.

3. The method of claim 1, wherein, The target storage data is page maintenance data of a client. Correspondingly, the method further comprises the following step: storing the target storage data in the database.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises the following steps: acquiring a data reading request, the data reading request comprising at least one target key name; determining a target key-value pair corresponding to each target key name in the cache according to the at least one target key name; acquiring a key value in each target key-value pair from all the determined target key-value pairs, the key value comprising at least one target field and an attribute value of each target field; feeding back a data reading response based on a pipeline mode of the cache, the data reading response comprising the key values in all the target key-value pairs.

5. The method of claim 4, wherein, Before the determining of the target key-value pair corresponding to each target key name in the cache according to the at least one target key name, the method further comprises the following step: storing the at least one target key name in a character string array. Correspondingly, the determining of the target key-value pair corresponding to each target key name in the cache according to the at least one target key name comprises the following steps: According to each target key name in the string array, the cache is queried in sequence to obtain a key value corresponding to each target key name.

6. A data processing apparatus, characterized by, The method comprises the following steps: An acquisition module is configured to acquire target storage data, which is data in a database or page maintenance data of a client. A processing module is configured to process the target storage data based on a preset key-value pair construction rule to obtain a cache key-value pair corresponding to the target storage data, wherein a key name of the cache key-value pair is used to indicate a data type of the target storage data, and a key value of the cache key-value pair is used to indicate a target storage field of the target storage data and an attribute value of the target storage field. A storage module is configured to store the key-value pair in a cache. The processing module is specifically configured to determine a data type of the target storage data according to business attribute information of the target storage data. According to the data type of the target storage data, an identification field and at least one class distinction field of the target storage data are determined. According to the identification field and the at least one class distinction field, a type identifier of the target storage data is determined, and the type identifier is used as the key name of the key-value pair. A target storage field frequently used in a financial business in the target storage data and an attribute value of the target storage field are determined. A class attribute name of the target storage field in a mapper mapping file of a data access layer is determined. According to the class attribute name and the attribute value of the target storage field, a key value of the cache key-value pair is constructed.

7. A server, characterized by The method comprises the following steps: A memory and a processor are provided. The memory is configured to store computer execution instructions executable on the processor. The processor executes the computer execution instructions to implement the method in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 5.

Images