A data processing method, device, apparatus, and computer-readable storage medium

Abstract

Embodiments of the present application disclose a data processing method, the method comprising: obtaining a query request for obtaining to-be-queried basic data; wherein the query request carries a data identifier of the to-be-queried basic data; determining a data type of the to-be-queried basic data based on the data identifier and parameters of the to-be-queried basic data; and obtaining the to-be-queried basic data from a memory and / or a target relational database based on the data type of the to-be-queried basic data. Embodiments of the present application also disclose a data processing apparatus, a device and a computer readable storage medium.

Description

Technical Field

[0001] This application relates to data processing technology in the field of computers, and more particularly to a data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] In Electron-based client application development, when dealing with fundamental data such as data dictionaries and product information that are frequently queried in business logic, different solutions are needed depending on the data volume and business requirements. The conventional approach is to request the fundamental data from the backend and cache it in memory when the application starts or a user logs in, then read it directly from memory on subsequent uses. However, since data retrieval is repeated every time the application is logged in or reopened, performance can be impacted when the data volume is large. To address this issue, the following optimization solution has emerged: after the initial retrieval of fundamental data, it is stored in a persistent database such as local storage or an indexed database. Upon subsequent logins, the cache is checked against the server based on local information to determine if new data needs to be retrieved. This solution avoids retrieving the full data on subsequent logins. This approach in related technologies can effectively handle the needs of most business scenarios. However, when the data to be cached is large, this data storage solution suffers from significant performance issues. Caching a large amount of fundamental data in memory leads to high memory consumption and impacts application stability. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a data processing method, apparatus, device, and computer-readable storage medium, which solves the significant performance issues in related technologies for basic data storage, avoids excessive memory consumption, and ensures application stability.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] A data processing method, the method comprising:

[0006] Obtain a query request for retrieving basic data to be queried; wherein the query request carries a data identifier of the basic data to be queried;

[0007] Based on the data identifier and the parameters of the basic data to be queried, the data type of the basic data to be queried is determined;

[0008] Based on the data type of the underlying data to be queried, the underlying data to be queried is obtained from memory and / or the target relational database.

[0009] In the above scheme, determining the data type of the basic data to be queried based on the data identifier and the parameters of the basic data to be queried includes:

[0010] The data volume and reading frequency of the basic data to be queried are determined based on the data identifier; wherein, the parameters include the data volume and the reading frequency;

[0011] If the data volume and the reading frequency meet the first data condition, the data type of the basic data to be queried is determined to be the first data type;

[0012] If the data volume and the reading frequency meet the second data condition, the data type of the basic data to be queried is determined to be the second data type;

[0013] If the data volume and the reading frequency meet the third data condition, the data type of the basic data to be queried is determined to be the third data type.

[0014] In the above scheme, obtaining the basic data to be queried from memory and / or the target relational database based on the data type of the basic data to be queried includes:

[0015] If the data type of the basic data to be queried is the first data type, the basic data to be queried is retrieved from the memory;

[0016] If the data type of the basic data to be queried is the second data type, the basic data to be queried is obtained from the target relational database;

[0017] If the data type of the basic data to be queried is the third data type, the basic data to be queried is obtained from the memory and / or the target relational database based on the storage status of the data in the memory;

[0018] Accordingly, the method further includes:

[0019] The basic data to be queried is transmitted to the first target business module.

[0020] In the above scheme, obtaining the basic data to be queried from the memory and / or the target relational database based on the storage status of the data in the memory includes:

[0021] If the basic data to be queried is not stored in the memory, the basic data to be queried is obtained from the target relational database and stored in the memory;

[0022] If the basic data to be queried is stored in the memory, the basic data to be queried is retrieved from the memory.

[0023] In the above scheme, before obtaining the query request for retrieving the underlying data to be queried, the following steps are also included:

[0024] If the target relational database exists and the basic data stored in the target relational database is valid, the changed basic data is obtained from the server based on the data information of the stored basic data;

[0025] Update the underlying data stored in the target relational database based on the changed underlying data;

[0026] If the target relational database does not exist or the stored basic data is invalid, obtain the initial basic data from the server;

[0027] The initial basic data is stored based on its data type.

[0028] In the above scheme, storing the initial basic data based on the data type of the initial basic data includes:

[0029] The data type of the initial basic data is determined based on the data volume of the initial basic data;

[0030] If the data type of the initial basic data is a fourth data type, the initial basic data is stored in the memory;

[0031] If the data type of the initial basic data is the fifth data type, the initial basic data is stored in the target relational database.

[0032] In the above scheme, determining the data type of the initial basic data based on the data volume of the initial basic data includes:

[0033] If the amount of the initial basic data meets the target data threshold, the data type of the initial basic data is determined to be the fourth data type.

[0034] If the amount of initial basic data does not meet the target data threshold, the data type of the initial basic data is determined to be the fifth data type.

[0035] The method in the above scheme further includes:

[0036] Receive a business request for obtaining business data; wherein the business request carries a business identifier;

[0037] Based on the business request, the target business data corresponding to the business identifier is obtained through the target data processing module;

[0038] The target business data is processed, and the processed target business data is stored in the target relational database;

[0039] The notification message used to notify that there is an update to the business data is carried in the target process message, and the second target business module is notified of the update to the business data through the target process message.

[0040] A data processing apparatus, the apparatus comprising:

[0041] The acquisition unit is used to acquire a query request for acquiring basic data to be queried; wherein the query request carries a data identifier of the basic data to be queried;

[0042] The determining unit is used to determine the data type of the basic data to be queried based on the data identifier and the parameters of the basic data to be queried;

[0043] The processing unit is used to retrieve the basic data to be queried from memory and the target relational database based on the data type of the basic data to be queried.

[0044] A data processing device, the device comprising: a processor, a memory, and a communication bus;

[0045] The communication bus is used to realize the communication connection between the processor and the memory;

[0046] The processor is used to execute the data processing program in the memory to implement the steps of the above-described data processing method.

[0047] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the data processing method described above.

[0048] The data processing method, apparatus, device, and computer-readable storage medium provided in the embodiments of this application can obtain a query request carrying a data identifier of the basic data to be queried for retrieving basic data to be queried. Based on the data identifier and parameters of the basic data to be queried, the data type of the basic data to be queried is determined. Then, based on the data type of the basic data to be queried, the basic data to be queried is retrieved from memory and / or a target relational database. In this way, after determining the data type of the basic data to be queried according to the data identifier and parameters of the basic data to be queried, the basic data to be queried can be retrieved from memory and / or a target relational database according to the different data types. Instead of retrieving all basic data from memory as in related technologies, this solves the obvious performance problems of basic data storage schemes in related technologies, avoids excessive memory consumption, and ensures application stability. Attached Figure Description

[0049] Figure 1 A schematic flowchart of a data processing method provided for an embodiment of this application;

[0050] Figure 2 A flowchart illustrating another data processing method provided for an embodiment of this application;

[0051] Figure 3 A schematic flowchart illustrating yet another data processing method provided in an embodiment of this application;

[0052] Figure 4 A flowchart illustrating the business data processing in a data processing method provided in another embodiment of this application;

[0053] Figure 5 A flowchart illustrating the business data processing in another data processing method provided in another embodiment of this application;

[0054] Figure 6 A schematic diagram of the structure of a data processing apparatus provided for an embodiment of this application;

[0055] Figure 7 This is a schematic diagram of the structure of a data processing device provided for an embodiment of this application. Detailed Implementation

[0056] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0057] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0058] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0059] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0060] This application provides a data processing method, which can be applied to a data processing device, as described above. Figure 1 As shown, the method includes the following steps:

[0061] Step 101: Obtain the query request used to retrieve the basic data to be queried.

[0062] The query request contains a data identifier for the underlying data to be queried.

[0063] In this embodiment, the basic data to be queried can refer to the basic data required by the application scenario in which the current business is located; basic data can refer to data that the business needs but is unrelated to the business. In one feasible implementation, basic data can refer to basic information, basic parameter configuration information, etc.; if the application scenario is a large-scale financial trading system, and the corresponding business is a large-scale specialized financial trading terminal project, then the basic data to be queried can include: various data dictionaries, more than 100,000 pieces of securities basic information, a large amount of basic parameter configuration information, etc.

[0064] The query request can be triggered when the application starts or when a user logs in. In one possible implementation, the query request can be for a large Electron-based desktop client application, meaning it can be generated when the application starts. It should be noted that the data identifier can be used to uniquely identify the underlying data to be queried.

[0065] Step 102: Determine the data type of the basic data to be queried based on the data identifier and the parameters of the basic data to be queried.

[0066] In this embodiment, after receiving a query request, the data processing device can determine the specific data referred to by the basic data to be queried based on the data identifier carried in the query request; furthermore, it can determine the data type of the basic data to be queried based on the parameters of the basic data to be queried. In other words, the basic data to be queried can be divided into different categories based on the parameters of the basic data to be queried.

[0067] The parameters of the underlying data to be queried can refer to inherent parameters of the underlying data itself; that is, these parameters can be considered as attribute parameters of the underlying data. In one feasible implementation, the parameters of the underlying data to be queried may include the data volume and reading frequency. In other words, the data processing device can determine the data type of the underlying data to be queried by analyzing the data volume and reading frequency.

[0068] Step 103: Based on the data type of the underlying data to be queried, retrieve the underlying data to be queried from memory and / or the target relational database.

[0069] In this context, the target relational database can refer to a lightweight database capable of persistently caching data. One feasible implementation is an SQLite database. Furthermore, SQLite is a relational database management system that adheres to ACID (atomicity, consistency, isolation, and durability). It is designed for embedded systems, aiming to provide local data storage for individual applications and devices, with very low resource consumption and extremely fast processing speed.

[0070] In other embodiments of this application, after determining the data type of the basic data to be queried, the data processing device can choose to retrieve the basic data from memory, from the target relational database, or from both memory and the target relational database, depending on the data type. It should be noted that the data processing method in this embodiment can significantly reduce the memory usage of the basic data; furthermore, because the persistent cache database can support multiple different processes sharing it, it effectively solves the problems of data sharing and memory duplication in multi-window business and application multi-instance startup modes.

[0071] The data processing method provided in the embodiments of this application can, after determining the data type of the basic data to be queried based on the data identifier and parameters of the basic data to be queried, select to retrieve the basic data to be queried from memory and / or the target relational database according to the different data types. Unlike related technologies, which retrieve all basic data from memory, this method solves the obvious performance problems of basic data storage schemes in related technologies, avoids excessive memory usage, and ensures the stability of the application.

[0072] Based on the foregoing embodiments, this application provides a data processing method, referring to... Figure 2 As shown, the method may include the following steps:

[0073] Step 201: The data processing device obtains a query request to retrieve the basic data to be queried.

[0074] The query request contains a data identifier for the underlying data to be queried.

[0075] Step 202: The data processing device determines the amount of basic data to be queried and the reading frequency based on the data identifier.

[0076] The parameters include data volume and reading frequency.

[0077] In this embodiment, after receiving a query request for basic data to be queried, the data processing device can first determine the specific data to be queried based on the data identifier of the basic data to be queried; then, it can obtain the attribute information of the specific data itself represented by the basic data to be queried, thereby obtaining the data volume and reading frequency of the basic data to be queried. The attribute information such as the data volume and reading frequency of the basic data to be queried can be pre-stored locally on the data processing device.

[0078] It should be noted that after step 202, you can choose to execute steps 203-204, or steps 205-206, or steps 207-208;

[0079] Step 203: If the data volume and reading frequency meet the first data condition, the data processing device determines the data type of the basic data to be queried as the first data type.

[0080] In this embodiment, the first data condition can refer to basic data that meets the condition, which occupies a relatively small amount of storage and is frequently used. In one feasible implementation, the first data condition can be that the data volume meets a first target data volume and the reading frequency meets a first target reading frequency; wherein, the first target data volume can be a data volume value preset based on historical data, and the first target reading frequency can be a reading frequency value preset based on historical data.

[0081] The first data condition, which is that the data volume and reading frequency meet the first data condition, can mean that the data volume is less than the first target data volume and the reading frequency is greater than the first target reading frequency. Furthermore, when the data volume is less than the first target data volume and the reading frequency is greater than the first target reading frequency, the data type of the basic data to be queried is the first data type that has a relatively small data volume and is frequently read and used.

[0082] Step 204: The data processing device retrieves the basic data to be queried from memory.

[0083] In the case where the data type of the basic data to be queried is the first data type with a relatively small data volume and is frequently read and used, it means that the basic data to be queried does not require much storage space, but the basic data needs to be read and used frequently, which means that the data of the first data type is stored in memory; at this time, the basic data to be queried can be read from memory.

[0084] Step 205: If the data volume and reading frequency meet the second data conditions, the data processing device determines that the data type of the basic data to be queried is the second data type.

[0085] In this embodiment, the second data condition can refer to basic data that meets the condition, which occupies a relatively large amount of storage and is not frequently used. In one feasible implementation, the second data condition can be that the data volume meets a second target data volume and the reading frequency meets a second target reading frequency. The second target data volume can be a pre-set data volume value based on historical data, and the second target reading frequency can be a pre-set reading frequency value based on historical data.

[0086] Specifically, satisfying the second data condition regarding data volume and reading frequency means that the data volume is greater than the second target data volume and the reading frequency is less than the second target reading frequency. Furthermore, when the data volume is greater than the second target data volume and the reading frequency is less than the second target reading frequency, the data type of the basic data to be queried is a second data type that is relatively large in volume and not frequently read. It should be noted that the second target data volume and the second target reading frequency are different from the first target data volume and reading frequency, respectively.

[0087] Step 206: The data processing device retrieves the basic data to be queried from the target relational database.

[0088] In cases where the data type of the underlying data to be queried is a second data type that is relatively large in volume and not frequently accessed, it indicates that the underlying data to be queried requires a large amount of storage space, but is not frequently accessed. This suggests that the second data type is stored in an SQLite database. In this case, the underlying data to be queried can be retrieved from the SQLite database.

[0089] Step 207: If the data volume and reading frequency meet the third data condition, the data processing device determines that the data type of the basic data to be queried is the third data type.

[0090] In this embodiment, the third data condition can refer to the basic data to be queried that occupies a relatively large amount of storage and some of the data is frequently used. In one feasible implementation, the third data condition can be that the data volume meets the second target data volume but the reading frequency does not meet the second target reading frequency.

[0091] The third data condition, where data volume and reading frequency meet the criteria, can mean that the data volume is greater than the second target data volume and the reading frequency of some data is greater than the second target reading frequency. Furthermore, when the data volume is greater than the second target data volume and the reading frequency of some data is greater than the second target reading frequency, the data type of the basic data to be queried is a third data type where the data volume is relatively large and some data will be read and used frequently. It should be noted that "the reading frequency of some data is greater than the second target reading frequency" can mean that a target proportion of data will be read and used frequently. This target proportion can be a threshold value set according to the specific application scenario.

[0092] Step 208: The data processing device retrieves the basic data to be queried from memory and / or the target relational database based on the storage status of the data in memory.

[0093] In cases where the type of basic data to be queried is a third data type with a large data volume and some data being frequently read and used, it indicates that the basic data to be queried requires a large storage space. However, since only some of the basic data is frequently read and used, it means that the third data type may be stored in memory or in an SQLite database. In this case, the basic data to be queried can be obtained from memory and / or the SQLite database based on the storage situation of the data in memory.

[0094] Step 208 can be implemented in the following way:

[0095] Step 208a: If the basic data to be queried is not stored in memory, the data processing device retrieves the basic data to be queried from the target relational database and stores the basic data to be queried in memory.

[0096] The storage status of data in memory refers to whether the basic data to be queried is stored in memory. If the basic data to be queried is not stored in memory, it can be read from the SQLite database. Furthermore, after successfully reading the basic data for the first time, it can be written into memory. This way, subsequent reads of the basic data can be directly retrieved from memory. Thus, a two-level data caching model is implemented, which, while addressing the issue of large amounts of data occupying memory, still provides the same high-performance read efficiency as the memory-based model.

[0097] Step 208b: If the basic data to be queried is stored in memory, the data processing device retrieves the basic data to be queried from memory.

[0098] In this embodiment of the application, if the basic data to be queried is stored in memory, it means that the basic data to be queried can be read from memory; at this time, the basic data to be queried can be read from memory, thereby reducing memory usage while achieving high-performance data query capability (i.e., achieving high-performance data reading for business).

[0099] It should be noted that step 209 can be executed after steps 204, 206, and 208;

[0100] Step 209: The data processing device transmits the basic data to be queried to the first target business module.

[0101] In this embodiment, the first target business module can refer to the business form corresponding to the application that requires the basic data to be queried. Furthermore, after reading the basic data to be queried, the data processing device can synchronously return the read basic data to the business form corresponding to the business, so that the business form can use the basic data to perform normal business operations. In one feasible implementation, the business form corresponding to the business can include multiple forms, such as business form 1, business form 2, ..., business form N.

[0102] In other embodiments of this application, reference is made to Figure 3 As shown, the method may further include the following steps before step 201:

[0103] Step 210: If a target relational database exists and the basic data stored in the target relational database is valid, the data processing device retrieves the changed basic data from the server based on the data information of the stored basic data.

[0104] When storing basic data, the specific storage scheme can be determined based on whether the data processing device has an SQLite database locally and whether the basic data stored in the SQLite database is valid.

[0105] In this embodiment, the data processing device first needs to check whether an SQLite database has been created locally. If an SQLite database has been created, it needs to check whether the basic data currently stored in the SQLite database is valid. If the basic data currently stored in the SQLite database is valid, it is only necessary to obtain the changed basic data, rather than obtaining all the basic data, which further avoids excessive memory usage and can improve the processing efficiency of basic data.

[0106] It should be noted that the data processing device can retrieve changed basic data from the server based on cached basic data information already stored in the local SQLite database. Here, the cached basic data information can refer to hash tags used to identify the data version of the basic data; and the changed basic data can be retrieved by the server based on the hash tags provided by the data processing device and then sent.

[0107] Step 211: The data processing device updates the basic data stored in the target relational database based on the changed basic data.

[0108] Upon receiving the changed basic data sent by the server, the changed basic data can be updated and written to the SQLite database. This updates the basic data currently stored in the SQLite database, ensuring that the SQLite database contains the latest version of the basic data required by the current business.

[0109] It should be noted that if the server determines that there is no changed underlying data based on the hash tag, then the data processing device does not need to obtain and preprocess any underlying data and can directly proceed to the next step of the user login process, thereby speeding up the login process.

[0110] Step 212: If the target relational database does not exist or the stored basic data is invalid, the data processing device obtains the initial basic data from the server.

[0111] In this embodiment of the application, if the data processing device detects that no SQLite database has been created locally, or if an SQLite database has been created locally but the basic data stored in the SQLite database is invalid, it means that the basic data stored in the SQLite database is no longer useful; at this time, the data processing device needs to obtain all the initial basic data required by the current business from the server.

[0112] Step 213: The data processing device stores the initial basic data based on the data type of the initial basic data.

[0113] In this embodiment, after receiving all the initial basic data required for the current service from the server, the data processing device can determine the amount of initial basic data and determine the data type of the initial basic data based on the amount of initial basic data; then, based on different data types, different storage schemes can be set for the initial basic data. Specifically, the initial basic data can be stored in memory or an SQLite database based on its different data types.

[0114] Step 213 can be achieved in the following way:

[0115] Step 213a: The data processing device determines the data type of the initial basic data based on the amount of initial basic data.

[0116] In this embodiment of the application, the data processing device can obtain the amount of initial basic data from the local machine; then analyze the amount of initial basic data to obtain the data type of the initial basic data.

[0117] It should be noted that step 213a can be achieved in the following way:

[0118] a1. If the amount of initial basic data meets the target data threshold, the data processing device determines the data type of the initial basic data to be the fourth data type.

[0119] The target data threshold can be a pre-set data volume value based on the current application scenario and historical data. If the initial basic data volume is greater than the target data threshold, then the data type of the initial basic data can be considered as the fourth data type with a relatively small data volume.

[0120] a2. If the amount of initial basic data does not meet the target data threshold, the data processing device determines the data type of the initial basic data to be the fifth data type.

[0121] If the amount of initial basic data is greater than the target data threshold, then the data type of the initial basic data can be considered to be the fifth data type with a relatively large amount of data.

[0122] Step 213b: If the data type of the initial basic data is the fourth data type, the data processing device stores the initial basic data in memory.

[0123] In this embodiment of the application, if the initial basic data is of the fourth data type with a relatively small amount of data, it means that the storage space required by the initial basic data is not particularly large, the degree of memory space usage is within the current acceptable range, and it will not affect the normal operation of other services supported on the memory. At this time, the initial basic data can be written into memory so that the current business window can query it at any time, thus ensuring data query efficiency.

[0124] Step 213c: If the data type of the initial basic data is the fifth data type, the data processing device stores the initial basic data into the target relational database.

[0125] In this embodiment of the application, if the data type of the initial basic data is the fifth data type with a large data volume, it means that the initial basic data requires a particularly large storage space, which has a significant impact on memory space and may affect the normal operation of other services supported in memory. In this case, the initial basic data can be written into the target relational database for subsequent business use and query at any time.

[0126] It should be noted that SQLite databases can support multiple independent business windows and multiple client instances sharing queries, meaning that multiple independent business windows and multiple application instances can share a single data cache; this effectively solves the problems of data sharing and memory duplication in multi-window business and application multi-instance startup modes.

[0127] In other embodiments of this application, reference is made to Figure 4 and 5 As shown, the method may further include the following steps:

[0128] Step 214: The data processing device receives a business request for obtaining business data.

[0129] The business request carries a business identifier.

[0130] In this embodiment of the application, the business request may be generated after the user performs an operation on the business form; that is, multiple independent business forms are running in their respective operating environments, and the user performs an operation on the business form during the business operation, thereby triggering the generation of a business request for obtaining business data.

[0131] Step 215: Based on the business request, the data processing device obtains the target business data corresponding to the business identifier through the target data processing module.

[0132] The target data processing module can be a module with an independent operating environment. In one feasible implementation, the target data processing module can refer to a data receiving and processing service, which establishes a connection with the server as an independent operating environment (an independent process). The server can be a remote server that provides services to the application.

[0133] After receiving the business request, such as Figure 5 As shown, in response to the business request, the data receiving and processing service can retrieve the target business data corresponding to the business identifier from the server.

[0134] Step 216: The data processing device processes the target business data and stores the processed target business data in the target relational database.

[0135] In this embodiment of the application, processing the target business data may refer to processing the format of the target business data; specifically, the target business data obtained from the server may be processed and converted according to the client's business requirements, and as follows: Figure 5 The diagram shows the target business data, after format processing and conversion, being written to the SQLite database. Here, the data processing device can refer to the client.

[0136] Step 217: The data processing device carries a notification message to indicate that the business data has been updated in the target process message, and notifies the second target business module that the business data has been updated through the target process message.

[0137] The target process message can be a message used for inter-process communication; in one feasible implementation, the target process message can refer to an inter-process communication (IPC) message. Specifically, after the target business data, which has undergone format processing and conversion, is written to the SQLite database, a notification message is carried in the IPC message, and this IPC message carrying the notification message is sent to, for example,... Figure 5 The business forms shown are 1 to M (i.e., the second target business module). After receiving the IPC message, business forms 1 to M will know that the business data stored in the SQLite database has been updated. They can then determine whether to respond to the update based on the IPC message content and their current business status. If they need to respond to the update, the business forms will actively query the latest business data from the SQLite database and update their current business status.

[0138] It should be noted that the SQLite database supports data retrieval using complex Structured Query Language (SQL) statements.

[0139] In other embodiments of this application, if business form 1 to business form M generates relevant business data, the generated business data can be submitted to the server through the business submission module.

[0140] Furthermore, this application's solution isolates the massive data processing logic by receiving and processing business data within a separate data receiving and processing service process and writing it to an SQLite database. This achieves a multi-process parallel processing mode, resolving the issues of increased memory usage and application lag caused by resource contention in business processes due to large-scale data processing. Simultaneously, by transmitting only change notifications via IPC messages, instead of transmitting the entire dataset across multiple processes as in related technologies, the amount of data exchanged between processes is significantly reduced, thereby greatly improving application performance and supporting the handling of massive amounts of business push data. Moreover, it avoids the performance overhead of large-scale data transfer between processes, effectively achieving data sharing in multi-window business scenarios.

[0141] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0142] The data processing method provided in the embodiments of this application can, after determining the data type of the basic data to be queried based on the data identifier and parameters of the basic data to be queried, select to retrieve the basic data to be queried from memory and / or the target relational database according to the different data types. Unlike related technologies, which retrieve all basic data from memory, this method solves the obvious performance problems of basic data storage schemes in related technologies, avoids excessive memory usage, and ensures application stability. At the same time, it can provide high-performance reading efficiency consistent with memory mode.

[0143] Based on the foregoing embodiments, this application provides a data processing apparatus that can be applied to... Figures 1-4 In the data processing method provided in the corresponding embodiment, refer to Figure 6 As shown, the data processing device 3 may include: an acquisition unit 31, a determination unit 32, and a processing unit 33, wherein:

[0144] Acquisition unit 31 is used to obtain a query request for obtaining the basic data to be queried;

[0145] The query request carries a data identifier for the underlying data to be queried.

[0146] The determining unit 32 is used to determine the data type of the basic data to be queried based on the data identifier and the parameters of the basic data to be queried.

[0147] Processing unit 33 is used to retrieve the underlying data to be queried from memory and the target relational database based on the data type of the underlying data to be queried.

[0148] In other embodiments of this application, the determining unit 32 is further configured to perform the following steps:

[0149] The amount of data and the reading frequency of the basic data to be queried are determined based on the data identifier; the parameters include the amount of data and the reading frequency.

[0150] If the data volume and reading frequency meet the first data condition, the data type of the basic data to be queried is determined to be the first data type;

[0151] If the data volume and reading frequency meet the second data condition, the data type of the basic data to be queried is determined to be the second data type;

[0152] If the data volume and reading frequency meet the third data condition, the data type of the basic data to be queried is determined to be the third data type.

[0153] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps:

[0154] If the data type of the underlying data to be queried is the first data type, retrieve the underlying data to be queried from memory;

[0155] If the data type of the underlying data to be queried is a second data type, retrieve the underlying data to be queried from the target relational database;

[0156] If the data type of the underlying data to be queried is a third data type, the underlying data to be queried will be obtained from memory and / or the target relational database based on the storage situation of the data in memory.

[0157] Correspondingly, the processing unit 33 is also used to transmit the basic data to be queried to the first target business module.

[0158] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps:

[0159] If the basic data to be queried is not stored in memory, retrieve the basic data to be queried from the target relational database and store the basic data to be queried in memory;

[0160] If the basic data to be queried is stored in memory, retrieve the basic data to be queried from memory.

[0161] In other embodiments of this application, the acquisition unit 31 is further configured to, if a target relational database exists and the basic data stored in the target relational database is valid, acquire the changed basic data from the server based on the data information of the stored basic data;

[0162] Processing unit 33 is also used to update the underlying data stored in the target relational database based on the changed underlying data;

[0163] The acquisition unit 31 is also used to acquire initial basic data from the server if the target relational database does not exist or the stored basic data is invalid;

[0164] The processing unit 33 is also used to store the initial basic data based on the data type of the initial basic data.

[0165] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps:

[0166] The data type of the initial basic data is determined based on the amount of initial basic data.

[0167] If the initial basic data is of the fourth data type, store the initial basic data in memory;

[0168] If the initial basic data is of the fifth data type, store the initial basic data in the target relational database.

[0169] In other embodiments of this application, the processing unit 33 is further configured to perform the following steps:

[0170] If the amount of initial basic data meets the target data threshold, the data type of the initial basic data is determined to be the fourth data type.

[0171] If the amount of initial basic data does not meet the target data threshold, the data type of the initial basic data is determined to be the fifth data type.

[0172] In other embodiments of this application, the acquisition unit 31 is further configured to receive a service request for acquiring service data; wherein the service request carries a service identifier;

[0173] The acquisition unit 31 is also used to acquire target business data corresponding to the business identifier through the target data processing module based on the business request;

[0174] Processing unit 33 is also used to process the target business data and store the processed target business data in the target relational database;

[0175] The processing unit 33 is also used to carry a notification message for notifying that the business data has been updated in the target process message, and to notify the second target business module that the business data has been updated through the target process message.

[0176] It should be noted that the specific implementation process of the steps performed by each unit in the embodiments of this application can be referred to Figures 1-4 The implementation process of the data processing method provided in the corresponding embodiments will not be described in detail here.

[0177] The data processing apparatus provided in the embodiments of this application can, after determining the data type of the basic data to be queried based on the data identifier and parameters of the basic data to be queried, select to retrieve the basic data to be queried from memory and / or the target relational database according to the different data types. Unlike related technologies, which retrieve all basic data from memory, this solves the obvious performance problems of basic data storage schemes in related technologies, avoids excessive memory usage, and ensures the stability of the application.

[0178] Based on the foregoing embodiments, embodiments of this application provide a data processing device that can be applied to... Figures 1-4 In the data processing method provided in the corresponding embodiment, refer to Figure 7 As shown, the data processing device 4 may include: a processor 41, a memory 42, and a communication bus 43, wherein:

[0179] Communication bus 43 is used to realize the communication connection between processor 41 and memory 42;

[0180] The processor 41 is used to execute the data processing program in the memory 42 to perform the following steps:

[0181] Obtain the query request used to retrieve the underlying data to be queried;

[0182] The query request carries a data identifier for the underlying data to be queried.

[0183] Based on the data identifier and the parameters of the underlying data to be queried, determine the data type of the underlying data to be queried;

[0184] Based on the data type of the underlying data to be queried, the underlying data to be queried is obtained from both the memory database and the target relational database.

[0185] In other embodiments of this application, the processor 41 is used to execute the data processing program in the memory 42 to determine the data type of the underlying data to be queried based on the data identifier and parameters of the underlying data to be queried, so as to implement the following steps:

[0186] The amount of data and the reading frequency of the basic data to be queried are determined based on the data identifier; the parameters include the amount of data and the reading frequency.

[0187] If the data volume and reading frequency meet the first data condition, the data type of the basic data to be queried is determined to be the first data type;

[0188] If the data volume and reading frequency meet the second data condition, the data type of the basic data to be queried is determined to be the second data type;

[0189] If the data volume and reading frequency meet the third data condition, the data type of the basic data to be queried is determined to be the third data type.

[0190] In other embodiments of this application, processor 41 is used to execute a data processing program in memory 42 based on the data type of the underlying data to be queried, and to obtain the underlying data to be queried from memory and a target relational database to perform the following steps:

[0191] If the data type of the underlying data to be queried is the first data type, retrieve the underlying data to be queried from memory;

[0192] If the data type of the underlying data to be queried is a second data type, retrieve the underlying data to be queried from the target relational database;

[0193] If the data type of the underlying data to be queried is a third data type, the underlying data to be queried will be obtained from memory and / or the target relational database based on the storage situation of the data in memory.

[0194] Accordingly, processor 41 is used to execute the data processing program in memory 42, and can also perform the following steps:

[0195] The basic data to be queried is transmitted to the first target business module.

[0196] In other embodiments of this application, the processor 41 is used to execute a data processing program in the memory 42 based on the storage status of data in memory, to obtain the basic data to be queried from memory and / or the target relational database, in order to implement the following steps:

[0197] If the basic data to be queried is not stored in memory, retrieve the basic data to be queried from the target relational database and store the basic data to be queried in memory;

[0198] If the basic data to be queried is stored in memory, retrieve the basic data to be queried from memory.

[0199] In other embodiments of this application, the processor 41 is used to execute a data processing program in the memory 42, and may also implement the following steps:

[0200] If a target relational database exists and the underlying data stored in the target relational database is valid, retrieve the changed underlying data from the server based on the data information of the stored underlying data;

[0201] Update the underlying data stored in the target relational database based on the changed underlying data;

[0202] If the target relational database does not exist or the stored underlying data is invalid, retrieve the initial underlying data from the server;

[0203] The initial basic data is stored based on the data type of the initial basic data.

[0204] In other embodiments of this application, processor 41 is used to execute a data processing program in memory 42 based on the data type of the initial underlying data, and to store the initial underlying data to implement the following steps:

[0205] The data type of the initial basic data is determined based on the amount of initial basic data.

[0206] If the initial basic data is of the fourth data type, store the initial basic data in memory;

[0207] If the initial basic data is of the fifth data type, store the initial basic data in the target relational database.

[0208] In other embodiments of this application, the processor 41 is used to execute a data processing program in the memory 42 to determine the data type of the initial basic data based on the amount of initial basic data, in order to implement the following steps:

[0209] If the amount of initial basic data meets the target data threshold, the data type of the initial basic data is determined to be the fourth data type.

[0210] If the amount of initial basic data does not meet the target data threshold, the data type of the initial basic data is determined to be the fifth data type.

[0211] In other embodiments of this application, the processor 41 is used to execute a data processing program in the memory 42, and may also implement the following steps:

[0212] Receive business requests for obtaining business data; the business requests carry a business identifier.

[0213] Based on the business request, the target business data corresponding to the business identifier is obtained through the target data processing module;

[0214] Process the target business data and store the processed target business data in the target relational database;

[0215] The notification message used to notify that there is an update to the business data is carried in the target process message, and the second target business module is notified of the update through the target process message.

[0216] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figures 1-4 The implementation process of the information data processing method provided in the corresponding embodiments will not be described in detail here.

[0217] The data processing device provided in this application embodiment can, after determining the data type of the basic data to be queried based on the data identifier and parameters of the basic data to be queried, select to retrieve the basic data to be queried from memory and / or the target relational database according to the different data types. Unlike related technologies, which retrieve all basic data from memory, this solves the obvious performance problems of basic data storage solutions in related technologies, avoids excessive memory usage, and ensures the stability of the application.

[0218] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement... Figures 1-4 The corresponding embodiments provide the steps of the data processing method.

[0219] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0220] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0221] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0222] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0223] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Obtain a query request for retrieving basic data to be queried; wherein the query request carries a data identifier of the basic data to be queried; Based on the data identifier and the parameters of the basic data to be queried, the data type of the basic data to be queried is determined; Based on the data type of the underlying data to be queried, the underlying data to be queried is obtained from memory and / or the target relational database; The step of determining the data type of the basic data to be queried based on the data identifier and the parameters of the basic data to be queried includes: The data volume and reading frequency of the basic data to be queried are determined based on the data identifier; wherein, the parameters include the data volume and reading frequency of the basic data to be queried; The data volume and reading frequency are analyzed and processed to obtain the data type of the basic data to be queried.

2. The method according to claim 1, characterized in that, The analysis and processing of the data volume and the reading frequency to obtain the data type of the basic data to be queried includes: If the data volume and the reading frequency meet the first data condition, the data type of the basic data to be queried is determined to be the first data type; If the data volume and the reading frequency meet the second data condition, the data type of the basic data to be queried is determined to be the second data type; If the data volume and the reading frequency meet the third data condition, the data type of the basic data to be queried is determined to be the third data type.

3. The method according to claim 2, characterized in that, The step of retrieving the basic data to be queried from memory and / or the target relational database based on the data type of the basic data to be queried includes: If the data type of the basic data to be queried is the first data type, the basic data to be queried is retrieved from the memory; If the data type of the basic data to be queried is the second data type, the basic data to be queried is obtained from the target relational database; If the data type of the basic data to be queried is the third data type, the basic data to be queried is obtained from the memory and / or the target relational database based on the storage status of the data in the memory; Accordingly, the method further includes: The basic data to be queried is transmitted to the first target business module.

4. The method according to claim 3, characterized in that, The step of retrieving the basic data to be queried from the memory and / or the target relational database based on the storage status of the data in the memory includes: If the basic data to be queried is not stored in the memory, the basic data to be queried is obtained from the target relational database and stored in the memory; If the basic data to be queried is stored in the memory, the basic data to be queried is retrieved from the memory.

5. The method according to claim 1, characterized in that, Before obtaining the query request for retrieving the underlying data to be queried, the process also includes: If the target relational database exists and the basic data stored in the target relational database is valid, the changed basic data is obtained from the server based on the data information of the stored basic data; Update the underlying data stored in the target relational database based on the changed underlying data; If the target relational database does not exist or the stored basic data is invalid, obtain the initial basic data from the server; The initial basic data is stored based on its data type.

6. The method according to claim 5, characterized in that, The data type based on the initial basic data, storing the initial basic data, includes: The data type of the initial basic data is determined based on the data volume of the initial basic data; If the data type of the initial basic data is a fourth data type, the initial basic data is stored in the memory; If the data type of the initial basic data is the fifth data type, the initial basic data is stored in the target relational database.

7. The method according to claim 6, characterized in that, Determining the data type of the initial basic data based on the data volume of the initial basic data includes: If the amount of the initial basic data meets the target data threshold, the data type of the initial basic data is determined to be the fourth data type. If the amount of initial basic data does not meet the target data threshold, the data type of the initial basic data is determined to be the fifth data type.

8. The method according to claim 1, characterized in that, The method further includes: Receive a business request for obtaining business data; wherein the business request carries a business identifier; Based on the business request, the target business data corresponding to the business identifier is obtained through the target data processing module; The target business data is processed, and the processed target business data is stored in the target relational database; The notification message used to notify that there is an update to the business data is carried in the target process message, and the second target business module is notified of the update to the business data through the target process message.

9. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire a query request for acquiring basic data to be queried; wherein the query request carries a data identifier of the basic data to be queried; The determining unit is configured to determine the parameters of the basic data to be queried based on the data identifier, and to analyze and process the data volume and reading frequency of the basic data to be queried to obtain the data type of the basic data to be queried; wherein, the parameters include the data volume and reading frequency of the basic data to be queried. The processing unit is used to retrieve the basic data to be queried from memory and the target relational database based on the data type of the basic data to be queried.

10. A data processing device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute a data processing program in the memory to implement the steps of the data processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the data processing method as described in any one of claims 1 to 8.

Images