Data processing method and device based on layered architecture, equipment and medium

Abstract

The invention relates to the technical field of data processing and the field of financial science and technology, and discloses a data processing method based on a layered architecture, and the method comprises the steps: dividing a data layer and a control layer, deploying a concurrent data processing module in the data layer, and configuring concurrent connection parameters with a distributed cache module; deploying a request control module in the control layer, and configuring control connection parameters with the distributed cache module; the control layer receives a request from an application component, determines whether the request is a common request or a high-concurrency request, accesses target information in the distributed cache module through an access interface of the control layer when processing the common request, and sends the request to the data layer when processing the high-concurrency request; and accessing the target information in the distributed cache module through the concurrent data processing module. According to the method, the data processing architecture is divided into the data layer and the control layer, so that the high-concurrency request can be directly processed through the data layer, the number of application components is greatly reduced, and resource waste is reduced.

Description

Technical Field

[0001] The present invention relates to the fields of data processing technology and financial technology, and in particular to a data processing method, device, equipment and storage medium based on a layered architecture. Background Art

[0002] In the financial field, as the digitalization process accelerates, financial institutions are facing an increasing number of concurrent data requests, especially during large-scale promotional activities, peak financial settlement periods, or market fluctuations, when a large number of user requests are concentrated, causing high concurrency pressure. Traditional data processing architectures usually handle concurrent user requests through business application components and middleware components. However, this model has the following shortcomings:

[0003] Business application components are mostly implemented using programming languages ​​such as Java, Go, and Python. Although these languages ​​are easy to program, they have relatively low performance and contain garbage collection mechanisms (GC), which perform poorly in high-concurrency scenarios. In contrast, middleware components are usually developed using high-performance, GC-free programming languages ​​such as C, C++, or Rust, and their processing power is usually more than 5 times higher than that of business application components. However, under the traditional architecture, each request must pass through the business application component and the middleware, which leads to a sharp increase in the number and load of business application components.

[0004] In financial business, promotional activities and market fluctuations are frequent, and financial institutions often need to temporarily expand business application components to cope with high concurrent requests. Although dynamic expansion can alleviate the pressure during peak periods, this strategy requires the preparation of a large number of physical machines in advance, resulting in a large demand for resource pools, high maintenance costs, and a large number of resources are idle at ordinary times, resulting in a huge waste of resources.

[0005] Since the traffic of each request needs to pass through the business application components and middleware in sequence, unnecessary traffic transfer and additional processing at the network level are caused. This architecture increases the delay in information access, affecting the user's access experience and system response efficiency.

[0006] In a high-concurrency environment, traditional architectures must share the load by adding a large number of business application components, which leads to a rapid expansion in the number of physical machines, making system management and maintenance more complicated, which is not conducive to effective cost control and the overall stability of the system. Summary of the invention

[0007] The main purpose of the present invention is to provide a data processing method, device, equipment and storage medium based on a layered architecture, aiming to solve the technical problems of the prior art in high concurrency scenarios, such as serious waste of resources and high system expansion costs due to the expansion of the number of business application components.

[0008] To achieve the above object, the present invention provides a data processing method based on a layered architecture, comprising:

[0009] Divide the data processing architecture into the data layer and the control layer;

[0010] In the data layer, deploy a concurrent data processing module, and configure concurrent connection parameters between the concurrent data processing module and the distributed cache module;

[0011] In the control layer, a request control module is deployed, and control connection parameters between the request control module and the distributed cache module are configured;

[0012] In the control layer, a request from an application component is received, and a type of the request is determined, where the type of the request includes a common request or a high-concurrency request;

[0013] If the request is a common request, access the target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer, and return the target information to the application component that initiated the request;

[0014] If the request is a high-concurrency request, the high-concurrency request is sent to the data layer, and the concurrent data processing module of the data layer accesses the target information from the distributed cache module according to the concurrent connection parameters, and returns the target information to the application component that initiated the request.

[0015] Furthermore, to achieve the above object, the present invention provides a data processing device based on a layered architecture, comprising:

[0016] An architecture management module, used to divide the data processing architecture into a data layer and a control layer;

[0017] A concurrent data processing module, used to deploy the concurrent data processing module in the data layer and configure concurrent connection parameters between the concurrent data processing module and the distributed cache module;

[0018] A request control module, used to deploy the request control module in the control layer and configure control connection parameters between the request control module and the distributed cache module;

[0019] A request identification module, used to receive a request from an application component in the control layer and determine the type of the request, wherein the type of the request includes a common request or a high-concurrency request;

[0020] A common request processing module, for accessing target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer if the request is a common request, and returning the target information to the application component that initiated the request;

[0021] A concurrent data processing module is used to send the high-concurrency request to the data layer if the request is a high-concurrency request, access the target information from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer, and return the target information to the application component that initiated the request.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a data processing program based on a layered architecture stored in the memory and executable on the processor, and the data processing program based on a layered architecture, when executed by the processor, implements the steps of the data processing method based on a layered architecture as described above.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a data processing program based on a layered architecture is stored, and when the data processing program based on a layered architecture is executed by a processor, the steps of the data processing method based on a layered architecture as described above are implemented.

[0024] Beneficial effects: The present invention relates to the fields of data processing technology and financial technology, and discloses a data processing method based on a layered architecture. By dividing the data layer and the control layer, a concurrent data processing module is deployed in the data layer, and concurrent connection parameters with the distributed cache module are configured; a request control module is deployed in the control layer, and control connection parameters with the distributed cache module are configured; the control layer receives requests from application components, determines whether the request type is a normal request or a high-concurrency request, and when processing a normal request, accesses the target information in the distributed cache module through the access interface of the control layer, and when processing a high-concurrency request, sends the request to the data layer, and accesses the target information in the distributed cache module through the concurrent data processing module. The present invention divides the data processing architecture into a data layer and a control layer, so that high-concurrency requests can be processed directly through the data layer, which greatly reduces the number of application components and reduces resource waste. At the same time, through modular management at different levels, the data access path is optimized, the delay is reduced, and the overall performance and scalability of the system are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1A schematic diagram of an application environment of a data processing method based on a layered architecture in an embodiment of the present invention;

[0027] Figure 2 It is a flowchart of an embodiment of a data processing method based on a layered architecture of the present invention;

[0028] Figure 3 A schematic diagram of functional modules of a preferred embodiment of a data processing device based on a layered architecture according to the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0030] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0032] The data processing method based on the layered architecture provided by the embodiment of the present invention can be applied in the following aspects: Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can divide the data processing architecture into a data layer and a control layer through the user terminal, deploy a concurrent data processing module in the data layer, and configure the concurrent connection parameters with the distributed cache module; deploy a request control module in the control layer, and configure the control connection parameters with the distributed cache module; the control layer receives a request from an application component, determines that the request type is a normal request or a high-concurrency request, and when processing a normal request, accesses the target information in the distributed cache module through the access interface of the control layer, and when processing a high-concurrency request, sends the request to the data layer, and accesses the target information in the distributed cache module through the concurrent data processing module. The present invention divides the data processing architecture into a data layer and a control layer, so that high-concurrency requests can be directly processed through the data layer, which greatly reduces the number of application components and reduces resource waste. At the same time, through modular management at different levels, the data access path is optimized, the delay is reduced, and the overall performance and scalability of the system are improved. Among them, the user terminal can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2This is a flow chart of an embodiment of a data processing method based on a layered architecture provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the data processing method based on the layered architecture proposed by the present invention includes the following steps:

[0035] S10, dividing the data processing architecture into a data layer and a control layer;

[0036] In this embodiment, the data processing architecture is designed to optimize data processing efficiency and simplify system management, and the responsibilities of data processing are clearly divided into two parts: the data layer and the control layer. The data layer is mainly responsible for data access, data caching, data calculation and large-scale concurrent processing in a high-concurrency environment. The control layer is mainly responsible for receiving, classifying and forwarding requests, as well as integration with upper-layer business logic.

[0037] The data layer focuses on processing high-frequency and high-concurrency data access requests, and improves data processing capabilities by using high-performance programming languages ​​and optimized concurrency control methods. The data layer is designed as an independent module to reduce the burden of application components directly processing high-concurrency requests.

[0038] The control layer is responsible for receiving requests from application components and classifying and scheduling requests. The design of the control layer can effectively reduce the complexity of direct data operations and optimize request flow through control and classification mechanisms.

[0039] The data layer and the control layer are deployed independently in the architecture, and the data is exchanged and synchronized with each other through high-speed network connections or shared distributed cache modules. This not only ensures the independence of the two layers, but also allows the two layers to be independently expanded or optimized according to the actual load conditions.

[0040] During the design phase of the system architecture, the responsibilities of the data layer and the control layer are clearly divided according to the frequency of data access, the complexity of business logic, and the concurrent load. High-concurrency processing tasks are concentrated in the data layer, and the reception, classification, and scheduling of requests are concentrated in the control layer. During the system deployment phase, according to the predetermined division strategy, the concurrent processing module of the data layer is deployed on a high-performance physical server or virtual machine; the request control module of the control layer is deployed on another set of servers to ensure that the two layers are physically separated. Define clear communication protocols and data interaction interfaces between the data layer and the control layer to ensure efficient collaboration and data consistency between the two layers. This can be achieved through technologies such as high-speed network connections, cache sharing, or RPC (remote procedure calls).

[0041] Example: In large-scale transaction processing scenarios in the financial field, such as in a bank's online payment system, it is usually necessary to process concurrent transaction requests from a large number of users. By dividing the data processing architecture into a data layer and a control layer, the control layer is responsible for receiving and parsing transaction requests from customers, and the data layer specializes in storing and querying transaction records. In this way, the control layer can efficiently classify and schedule requests, and hand over high-concurrency requests to the data layer for processing, thereby reducing system latency and improving data processing throughput.

[0042] By dividing the data processing architecture into the data layer and the control layer, the module design of the system can be effectively simplified, and the efficiency of data processing and the scalability of the system can be improved. The data layer focuses on data access in high-concurrency scenarios, reducing the burden of application components directly handling high concurrency; the control layer is responsible for receiving and scheduling requests, reducing system delays caused by complex data processing. This layered architecture can reduce the number of components, optimize resource utilization, and make system expansion and maintenance more flexible.

[0043] S20, deploying a concurrent data processing module in the data layer, and configuring concurrent connection parameters between the concurrent data processing module and the distributed cache module;

[0044] In this embodiment, the concurrent data processing module is deployed in the data layer mainly to handle high-concurrency data requests. The core of the module design is to improve the efficiency of data access and avoid system bottlenecks and performance degradation caused by high-concurrency access. The concurrent data processing module is generally written in a high-performance language (such as C++, Rust) to optimize concurrent data access capabilities.

[0045] Concurrent connection parameters include multiple configuration items, such as the size of the connection pool, the timeout period for each connection, the maximum number of concurrent connections, and the number of retries, etc. The configuration of these parameters is intended to optimize the concurrent connections between the module and the distributed cache module, and improve the data reading and writing efficiency and response speed.

[0046] The concurrent data processing module needs to establish multiple concurrent connections with the distributed cache module in order to process a large number of parallel data access requests in a high-concurrency environment. The purpose of configuring the connection is to control the size of the connection pool, the life cycle of each connection (configured by the timeout time), and the retry strategy when the connection fails.

[0047] First, install the concurrent data processing module on the server or virtual machine of the data layer. The installation of this module can be completed through automated deployment tools, such as using containerization technology (such as Docker) to deploy the module to different nodes to ensure the scalability of the module. After the installation is complete, set the connection parameters for the concurrent data processing module through the configuration file or management platform. Typical configuration items include:

[0048] Connection pool size: used to define the maximum number of concurrent connections between the data layer and the distributed cache module.

[0049] Timeout: used to set the maximum survival time of each connection to prevent the connection resources from being occupied for a long time.

[0050] Retry times: used to set the number of retries and strategies after a connection failure or timeout.

[0051] When the concurrent data processing module is initialized, it automatically establishes multiple connections with the distributed cache module according to the preset concurrent connection parameters, and manages these connection resources in a unified manner. The module will respond to the ever-changing concurrent load by dynamically allocating and recycling connection resources to ensure data processing efficiency in a high-concurrency environment.

[0052] Example description: In financial data processing, such as in a stock trading system, when a large number of users submit buy and sell orders at the same time, the concurrent data processing module can quickly process concurrent order requests in the data layer. By configuring 1,000 concurrent connections and setting a reasonable timeout and retry strategy, the module can ensure that each order request can be quickly received and processed, thereby reducing the user's waiting time and improving the system's response speed and processing capabilities.

[0053] By deploying concurrent data processing modules in the data layer and properly configuring concurrent connection parameters, the data processing efficiency in a high-concurrency environment can be effectively improved. Reasonable management and dynamic adjustment mechanisms of the connection pool ensure the stability and efficiency of the system when processing a large number of concurrent requests. The optimized connection strategy with the distributed cache module can reduce data access delays and improve the overall data processing throughput.

[0054] S30, deploying a request control module in the control layer, and configuring control connection parameters between the request control module and the distributed cache module;

[0055] In this embodiment, the main function of the control layer is to receive, classify and schedule requests. The request control module is a core module deployed in the control layer, which is used to monitor requests from various application components. The module parses and classifies requests through preset rules or algorithms, and sends different types of requests to appropriate processing modules for processing.

[0056] The request control module needs to establish a connection with the distributed cache module so that it can directly obtain data from the distributed cache when processing ordinary requests. The control connection parameters include the size of the connection pool, the maximum number of concurrent connections for each connection, the timeout period, etc. The configuration of these parameters is mainly aimed at the processing requirements of ordinary requests, focusing on the stability of the connection and resource utilization.

[0057] The request control module manages the connection with the distributed cache module according to the set control connection parameters. Its purpose is to provide efficient and stable connection management in the scenario of ordinary requests, while avoiding excessive pressure on the access to the distributed cache module.

[0058] Install the request control module on the physical server or virtual machine of the control layer. The module is mainly deployed through the standard service installation method. Usually, the module installation can be completed through automation tools (such as Ansible, Kubernetes, etc.) to simplify the deployment process. After the deployment is completed, set the control connection parameters for the request control module through the configuration file or system management platform. These parameters include:

[0059] Connection pool size: specifies the maximum number of connections between the control layer and the distributed cache module to handle concurrent access to common requests.

[0060] Maximum number of concurrent requests: Limits the number of common requests processed simultaneously in the control layer to prevent request overload from affecting the stability of the control layer.

[0061] Timeout: Set the maximum survival time of each connection in the control layer to ensure the stability of the connection and timely recovery.

[0062] When the request control module is initialized, it establishes multiple stable connections with the distributed cache module according to the preset control connection parameters. The module will dynamically allocate and manage these connection resources according to the access requirements of ordinary requests, thereby achieving efficient data access in the processing of requests.

[0063] Example: In the banking settlement system in the financial field, a large number of common query requests are usually processed, such as account balance query or transaction record query. By deploying the request control module in the control layer and configuring the control connection parameters with the distributed cache module, these common requests can be responded to quickly. The reasonable setting of the control connection pool (such as 10 connection pools), combined with the appropriate timeout time and maximum concurrency configuration, can ensure that system resources are efficiently used in the process of processing common requests and avoid response delays caused by unstable connections.

[0064] By deploying the request control module in the control layer and properly configuring the control connection parameters, the processing efficiency and stability of common requests can be significantly improved. The request control module focuses on receiving and managing common requests, and can flexibly select appropriate connection resources according to the request type, thereby reducing the impact of the control layer on high-concurrency request processing. In addition, through parameterized configuration, flexible connection management can be performed for different business scenarios, improving the scalability and reliability of the control layer.

[0065] S40, in the control layer, receiving a request from an application component and determining a type of the request, where the type of the request includes a common request or a high-concurrency request;

[0066] In this embodiment, one of the core responsibilities of the control layer is to receive various requests from application components. Application components are the interface between the client and the system. They convert user operations (such as query, submission, update, etc.) into system requests and send them to the control layer. The request control module of the control layer receives these requests by monitoring network ports or API interfaces, and extracts key information through preliminary analysis.

[0067] After receiving the request, the request control module will parse the content and parameters of the request. The parsed content includes the type of request (such as read operation, write operation), the priority of the request, the source information of the request, etc. The purpose of this step is to obtain enough information to prepare for the subsequent request classification.

[0068] The classification of requests is mainly based on preset rules or policies. According to the content and characteristics of the request, the control layer will classify the request into "normal request" or "high concurrency request". Normal request usually refers to the request that requires less computing resources and has a lower priority; high concurrency request involves a large amount of data processing and high priority. The classification of requests is usually implemented using rule engines or simple logical judgments.

[0069] The classification of requests is usually based on the following factors: the type of request (such as query, write, etc.), the amount of data in the request body, the priority of the request (such as VIP user request), etc. Through these classification criteria, the system can dynamically determine the request type in order to select the appropriate processing path.

[0070] For example, the request classification rules can be set based on the size of the request data:

[0071] Ordinary request: The data volume of a single request is small, usually from a few hundred bytes to a few thousand bytes. For example, the data volume of requests such as checking account balances and obtaining basic user information is small.

[0072] High-concurrency requests: The amount of data in a single request is large, which may reach hundreds of KB or even several MB. For example, querying a large number of transaction records, obtaining detailed statistical analysis data, etc.

[0073] A single request with a data volume exceeding 50 KB can be considered a high-concurrency request.

[0074] Request classification rules can be set based on the request processing time:

[0075] Ordinary requests: The expected processing time is short, usually between tens of milliseconds and hundreds of milliseconds. For example, requests to query a user's basic information or account balance.

[0076] High-concurrency requests: These requests take a long time to process, usually more than 500 milliseconds. For example, they process large amounts of order data or perform complex calculations and data aggregation operations.

[0077] A request that takes more than 500 milliseconds to process a single request can be considered a high-concurrency request.

[0078] The request classification rules can also be set based on the number of concurrent requests:

[0079] Ordinary requests: The number of concurrent requests is relatively low and can be stabilized within a few hundred. For example, users' scattered query operations or basic data requests during non-peak hours.

[0080] High concurrent requests: The number of concurrent requests is high, reaching thousands or tens of thousands. For example, peak requests during large promotions or market fluctuations.

[0081] Requests with more than 500 concurrent requests can be considered high-concurrency requests.

[0082] Request classification rules can also be set based on the priority and source of the request:

[0083] Normal requests: Requests are of lower priority and are usually from ordinary users or regular automated tasks. For example, daily account check tasks.

[0084] High-concurrency requests: Requests with a higher priority may come from batch processing tasks, large-scale promotions, or batch requests from VIP users.

[0085] Batch requests from specific user groups (such as VIP users) or specific events (such as promotions) can be considered high-concurrency requests.

[0086] In actual applications, the system can comprehensively determine whether a request is a high-concurrency request based on a combination of multiple quantitative criteria. The following are some examples of rules:

[0087] Rule 1: If the amount of data in a single request exceeds 50KB, it is considered a high-concurrency request.

[0088] Rule 2: If the processing time of a single request exceeds 500 milliseconds, it is considered a high-concurrency request.

[0089] Rule 3: If the number of concurrent requests at the same time exceeds 500, it is considered a high-concurrency request.

[0090] Rule 4: If the source of the request is a batch task or a large batch request from a VIP user, it is considered a high-concurrency request.

[0091] The request control module listens to the network interface or open API endpoint at startup to receive requests from application components in real time. When a request arrives, the control module first confirms the legitimacy of its source and puts it into the request queue. When the request enters the control module, the system extracts the core parameters of the request by parsing the information in the request header and request body. These parameters include operation type, request data volume, priority flag, etc. After parsing, this information is passed as input to the classification logic. The request control module classifies the request into ordinary requests or high-concurrency requests based on the parsed parameters and preset classification rules (such as judging the size of the request data volume, the complexity of the operation type, etc.). The classification result will determine the processing path of subsequent requests.

[0092] Example: In the customer transaction query system in the financial field, the user's query requests can be divided into ordinary account balance queries and transaction detail queries that require processing a large amount of data. Through the request control module in the control layer, the system can classify the received requests, treating account balance queries as ordinary requests and transaction detail queries as high-concurrency requests. Through this classification, ordinary requests can be quickly responded to directly in the control layer, while high-concurrency requests are forwarded to the data layer for parallel processing, thereby improving the system's response speed and processing efficiency.

[0093] By receiving and parsing requests in the control layer and classifying the requests, the system's processing efficiency for different types of requests can be significantly improved. Through dynamic classification, the priority and processing path of requests can be flexibly adjusted according to the current load status of the system, ensuring the rational use of system resources and avoiding system congestion in high concurrency situations. At the same time, the classification processing method also allows high-concurrency requests to be processed exclusively by the data layer, while ordinary requests can be completed directly in the control layer.

[0094] S50, if the request is a common request, access the target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer, and return the target information to the application component that initiated the request;

[0095] In this embodiment, after receiving the request, the request control module in the control layer determines the type of the current request as a common request according to a preset rule or classification algorithm. Common requests usually include requests with small data volume and low computational burden, such as account balance query, basic information query, etc.

[0096] The request control module calls the configured standardized access interface to process common requests based on the judgment results. These access interfaces are usually directly connected to the distributed cache module to quickly obtain target data. The design of the interface call follows the principles of simplicity and efficiency to reduce unnecessary processing steps and delays.

[0097] The control connection parameters are used to control the connection management strategy of the access interface, such as the size of the connection pool, the timeout of a single connection, the maximum number of retries, etc. When calling the access interface, the request control module will refer to these parameters to dynamically allocate and manage the connection with the distributed cache module.

[0098] Through the access interface, the request control module establishes a connection with the distributed cache module and queries the target information from the cache according to the content of the common request. The distributed cache module usually stores high-frequency access data to improve data access efficiency and response speed.

[0099] When the target information is successfully obtained, the control module will send the target information back to the application component that initiated the request according to the return path and format in the request, completing the entire normal request processing flow.

[0100] In the request control module, the system determines whether the current request is a normal request based on preset rules. If it is a normal request, the system will select an appropriate access interface for processing. The control module calls the pre-configured access interface according to the type of normal request. Before calling the access interface, it allocates an idle connection from the connection pool according to the control connection parameters, and sets the timeout time and retry strategy. The access interface interacts with the distributed cache module through the allocated connection to query the target information. If the target information exists in the cache, it returns immediately. If the target information does not exist in the cache, further processing may be required according to the configuration. After successfully obtaining the target information, the access interface sends the data to the application component that initiated the request through a predetermined return path. After the data is returned, the control module releases the allocated connection and returns it to the connection pool to serve the next request.

[0101] Example: In the credit card bill query system in the financial field, users usually query basic information such as the current account balance and credit limit. These query operations have a small amount of data and a short processing time, and are ordinary requests. Through the request control module of the control layer, the system can quickly identify and classify these requests, and query relevant information from the distributed cache module through a standardized access interface, and quickly return the data required by the user, thereby significantly improving the user experience and reducing the system load pressure.

[0102] By providing a standardized access interface for common requests, the system can significantly reduce the processing overhead of common requests and improve the response speed. Through reasonable connection parameter configuration, the control module can flexibly manage the connection resources with the distributed cache module, avoiding the waste of connection resources in common request scenarios. At the same time, through the standardized design of layered processing and access interfaces, the stability and scalability of the system can be improved.

[0103] S60: If the request is a high-concurrency request, the high-concurrency request is sent to the data layer, and the target information is accessed from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer, and the target information is returned to the application component that initiated the request.

[0104] In this embodiment, in the control layer, the request control module determines and identifies whether the current request is a high-concurrency request based on the request characteristics, data volume, user priority or current system load status. High-concurrency requests usually involve operations with large data volumes, high concurrency numbers and long processing times.

[0105] After determining that the request is a high-concurrency request, the request control module will send the request to the concurrent data processing module of the data layer through a specified network path or a dedicated interface. The data layer focuses on processing a large number of concurrent requests and is therefore designed to support high throughput and parallel data processing.

[0106] After receiving high-concurrency requests, the concurrent data processing module of the data layer processes them according to the preset concurrent connection parameters. These parameters include the size of the connection pool, timeout period, maximum number of concurrent connections, etc. Based on these parameters, the concurrent data processing module allocates appropriate connections from the initialized connection pool to interact with the distributed cache module.

[0107] The concurrent data processing module establishes a parallel connection channel with the distributed cache module through the allocated connection. The module will obtain the target information from the cache using an efficient data access algorithm based on the specific content of the request. If the target information is not in the cache, subsequent data acquisition strategies may be triggered.

[0108] When the target information is successfully obtained, the data layer will send the target information back to the application component that initiated the request according to the return path and data format in the request. At the same time, the concurrent data processing module will release the used connection resources and return them to the connection pool to ensure the recycling of resources.

[0109] After the request control module determines that the current request is a high-concurrency request through the rule engine or the preset classification rules, it forwards the request to the concurrent data processing module of the data layer using a dedicated high-concurrency interface. When forwarding, the module will attach the key parameters and return path information of the request. After receiving the high-concurrency request, the concurrent data processing module allocates appropriate connection resources from the connection pool according to the concurrent connection parameters. The module will decide how many concurrent connections to use based on the content and data volume of the request to optimize the processing efficiency of concurrent requests. The concurrent data processing module uses the allocated connection and an efficient data access strategy to read the target information from the distributed cache module. In high-concurrency scenarios, the module may enable optimization technologies such as parallel reading and batch reading to ensure the stability and efficiency of data access. When the data access is completed, the module returns the target information to the application component that initiated the request according to the path in the request. After that, the module will release and recycle the used connection resources to ensure the stability and availability of the connection pool.

[0110] Example: In the online payment system in the financial field, whenever a large-scale promotion is carried out or a large transaction is processed, a large number of high-concurrency requests will appear. The control layer can quickly identify these high-concurrency requests and forward them to the data layer for processing. The concurrent data processing module of the data layer quickly allocates connection resources according to the configured concurrent connection parameters and establishes an efficient data channel with the distributed cache module, thereby processing a large number of concurrent transaction requests in a short time, reducing the user's waiting time and improving the overall response speed of the system.

[0111] By forwarding high-concurrency requests to the data layer for processing, the load of the control layer can be effectively dispersed and the scalability of the system can be improved. At the same time, the concurrent data processing module can flexibly manage the connection resources between the distributed cache module according to the preset concurrent connection parameters, ensuring the stability and high efficiency of data access in a high-concurrency environment. Through layered architecture design and optimal scheduling of resources, the present invention can significantly reduce the response delay of the system and improve the overall throughput of concurrent data processing.

[0112] The present invention relates to the fields of data processing technology and financial technology, and discloses a data processing method based on a layered architecture. By dividing the data layer and the control layer, a concurrent data processing module is deployed in the data layer, and concurrent connection parameters with the distributed cache module are configured; a request control module is deployed in the control layer, and control connection parameters with the distributed cache module are configured; the control layer receives a request from an application component, determines that the request type is a common request or a high-concurrency request, and when processing a common request, accesses the target information in the distributed cache module through the access interface of the control layer, and when processing a high-concurrency request, sends the request to the data layer, and accesses the target information in the distributed cache module through the concurrent data processing module. The present invention divides the data processing architecture into a data layer and a control layer, so that high-concurrency requests can be directly processed through the data layer, which greatly reduces the number of application components and reduces resource waste. At the same time, through modular management at different levels, the data access path is optimized, the delay is reduced, and the overall performance and scalability of the system are improved.

[0113] In one embodiment, in the above S60, after accessing the target information from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer, the method further includes:

[0114] S701, when the concurrent data processing module fails to successfully access the target information from the distributed cache module, the target information is acquired from the database through the concurrent data processing module;

[0115] S702, caching the target information into the distributed cache module;

[0116] S703: Return the target information to the application component that initiated the request.

[0117] In this embodiment, the concurrent data processing module of the data layer is used to directly obtain the target information from the distributed cache module according to the concurrent connection parameters (including connection pool size, timeout time, number of retries, etc.). The concurrent data processing module allocates connections from the preconfigured connection pool, establishes a high-speed connection channel with the distributed cache module, and quickly queries the target information through parallel access technology.

[0118] When the concurrent data processing module attempts to obtain target information from the distributed cache module, access failure may occur due to cache miss, data expiration, or connection failure. At this time, the concurrent data processing module needs to be able to detect these abnormal situations and take remedial measures.

[0119] When access to the distributed cache module fails, the concurrent data processing module switches to the backup database data source and requeries the target information from the database. The data source in the database is considered authoritative and persistent, so when the cache misses, the module directly obtains the latest target information from the database.

[0120] After successfully obtaining the target information from the database, the concurrent data processing module will write this information into the distributed cache module so that it can be directly obtained from the cache the next time it is accessed, thereby improving the processing efficiency of subsequent requests. This step can avoid querying data from the database multiple times and reduce the load pressure on the database.

[0121] Finally, the concurrent data processing module returns the successfully acquired target information to the application component that initiated the request along the preset path. When returning, efficient transmission protocols or compression technologies can be used to reduce data transmission delays and improve user experience.

[0122] For example, the concurrent data processing module attempts to query the target information from the distributed cache module through the assigned connection. During the access process, the module will set a timeout and control the number of retries based on the connection parameters. If the data is not successfully obtained within the specified time, it is deemed to be an access failure. When cache access fails, the module immediately switches to the database connection and re-executes the query operation from the database based on the requested target information. The database access method usually uses SQL or other optimized query technologies to reduce query time. After successfully obtaining the target information, the module will update the obtained data to the distributed cache module and set the cache time or expiration policy of the data to ensure the timeliness and validity of the cached data. Then, the target information is sent to the initiator of the request through the configured return path.

[0123] Through the above steps, this embodiment can ensure that in a high-concurrency environment, even if the distributed cache fails to provide valid data, the latest target information can be obtained in a timely manner through the database. At the same time, writing the successfully obtained target information back to the distributed cache module can improve the cache hit rate of subsequent requests, reduce the query pressure of the database, and improve the overall data access efficiency and system response speed.

[0124] In one embodiment, the above S20 includes:

[0125] S201, installing the concurrent data processing module on the physical server or virtual machine of the data layer;

[0126] S202, in the concurrent data processing module, setting concurrent connection parameters with the distributed cache module, the concurrent connection parameters including connection pool size, timeout time and number of retries;

[0127] S203: Initialize a connection pool according to the connection pool size, and pre-allocate available connections with the distributed cache module in the connection pool.

[0128] In this embodiment, a concurrent data processing module is installed on a hardware resource (such as a physical server or a virtual machine) of the data layer. The installation of the module includes selecting appropriate hardware or virtualized resources to ensure the computing power and network bandwidth for processing high concurrent requests.

[0129] In the concurrent data processing module, you need to set the connection parameters with the distributed cache module according to the actual concurrent access requirements. These parameters include:

[0130] Connection pool size: indicates the maximum number of concurrent connections allowed between the module and the distributed cache module.

[0131] Timeout: Indicates the maximum survival time of each connection, which is used to prevent the connection from occupying resources for too long.

[0132] Retry times: indicates the maximum number of retries when the connection fails or times out, used to ensure the stability of the connection.

[0133] The connection pool is used to manage and allocate multiple connection resources between the distributed cache module. The process of initializing the connection pool includes creating and managing these connection resources according to the set connection pool size.

[0134] After initialization, the connection pool will pre-establish connections with the distributed cache module according to the configured size and manage these connections as available resources. The purpose of pre-allocation is to quickly allocate idle connections when processing data access requests and improve the efficiency of concurrent processing.

[0135] Install the service program of the concurrent data processing module on the physical server or virtual machine of the data layer. The installation process can be implemented through automated deployment tools (such as Ansible, Kubernetes, etc.) to ensure that the service instance of the module can run on each physical server or virtual machine. Set the concurrent connection parameters with the distributed cache module in the module configuration file or through the management platform. The connection pool size determines the maximum number of connections allowed; the timeout is set to a reasonable length of time (such as 30 seconds) to avoid long-term connection occupation; the number of retries is set to 1-3 times to deal with connection failures. When the module starts, it is initialized according to the set connection pool size and multiple connection objects with the distributed cache module are created. The operation of initializing the connection pool usually includes establishing multiple idle connections with the cache module in advance so that it can respond quickly when processing requests. After the initialization is completed, the connection pool manages these connection resources in memory. Each connection object is marked as "available" during the initialization phase to ensure that subsequent request processing can quickly obtain idle connections.

[0136] This embodiment can effectively improve the efficiency of data access and the stability of connection by deploying concurrent data processing modules in the data layer and reasonably setting concurrent connection parameters with the distributed cache module. The pre-allocated idle connection pool ensures fast response capabilities in a high-concurrency environment. At the same time, through a reasonable timeout and retry strategy, the stability of the connection can be improved, and system interruptions and resource waste can be reduced.

[0137] In one embodiment, in the above S60, accessing the target information from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer includes:

[0138] S601, determining an idle connection from a pre-initialized connection pool through the concurrent data processing module, and performing a data access operation in the distributed cache module through the idle connection;

[0139] S602, monitoring the running state of the current connection through the concurrent data processing module, and determining whether the current connection has timed out according to the timeout time in the concurrent connection parameter;

[0140] S603, if the concurrent data processing module detects that a connection timeout occurs in the current connection, the current connection is released, and a connection with the distributed cache module is reestablished according to the number of retries in the concurrent connection parameter;

[0141] S604, when the target information is successfully accessed, the concurrent data processing module releases and returns the currently used connection to the connection pool.

[0142] In this embodiment, when processing high concurrent requests, the concurrent data processing module searches and determines an idle connection from the initialized connection pool. Each connection in the connection pool has been initialized when the module is started, and a connection status identifier is maintained to indicate whether the connection is "idle" or "in use".

[0143] After finding an idle connection, the concurrent data processing module establishes an access channel with the distributed cache module through the connection and performs data access operations. Data access operations generally include reading or writing target information in the cache.

[0144] During the data access process, the concurrent data processing module needs to continuously monitor the operating status of the current connection. The monitoring of the operating status mainly involves the effectiveness, stability and response time of the connection. The purpose of monitoring is to be able to handle abnormal situations in a timely manner when they are detected.

[0145] The module determines whether the current connection completes the access operation within the preset time according to the timeout time set in the concurrent connection parameters. If the set time threshold is exceeded, it is considered that the connection has timed out.

[0146] If the module detects a connection timeout, it will immediately release the current connection to avoid occupying limited connection resources. Then, according to the number of retries in the concurrent connection parameters, it will try to re-establish the connection with the distributed cache module to ensure the continuity and stability of data access.

[0147] When data access is successfully completed, the concurrent data processing module releases the currently used connection and marks it as "idle" so that the next request can continue to use the connection. The release process usually includes cleaning up the connection status and resources to ensure the stability and effective operation of the connection pool.

[0148] The concurrent data processing module determines the idle connection based on the connection status mark by searching the connection object in the connection pool. In terms of implementation, an available connection resource can be quickly obtained by traversing the connection pool or using a connection queue. After determining the idle connection, the module sends a data access request through the connection and returns the target information based on the response of the distributed cache module. During the access process, the module records the usage status of the connection so that it can be processed in case of timeout or failure. The module regularly checks the status of each connection through a timer or monitoring thread. If a connection exceeds the preset timeout, the module will immediately release the connection to prevent it from occupying resources. At the same time, according to the set number of retries, the connection will be reallocated and access will be attempted again. After successfully completing the data access, the module will re-mark the status of the current connection as "idle" and return it to the connection pool. The connection release process also includes resetting the connection object and cleaning up resources to ensure the stability of the connection pool.

[0149] This embodiment selects idle connections from a pre-initialized connection pool and monitors the status of the connection in real time during the access process. The system can significantly improve the efficiency and stability of high-concurrency data access. At the same time, through reasonable timeout and retry strategies, the long-term occupation of connection resources and the interruption of data access can be effectively avoided. In addition, the stability and continuity of the connection pool can be guaranteed by releasing and recycling resources after successful access.

[0150] In one embodiment, the above S40 includes:

[0151] S401, monitoring and receiving a request from an application component through a request control module in the control layer;

[0152] S402, parsing the request and extracting parameter information in the request header and the request body from the request;

[0153] S403: Classify the request into a common request or a high-concurrency request according to the parameter information and by using a preset classification rule.

[0154] In this embodiment, after the request control module is started in the control layer, it will continue to monitor through the network port or a specific API interface. The purpose of monitoring is to receive requests from different application components in real time and to process requests from different sources to ensure service continuity.

[0155] After receiving the request from the application component, the request control module needs to parse the content of the request. The request usually includes a request header (such as request type, request source, user information, etc.) and a request body (such as specific query parameters or data operation content). The purpose of parsing is to extract key information from the request to determine the type of request.

[0156] The request control module judges and classifies requests according to preset classification rules. Classification rules can be based on the content of the request, the source of the request, the amount of data requested, priority, etc. Through the judgment of these rules, the module can classify requests as "normal requests" or "high concurrency requests".

[0157] Classification rules are configured in advance and adjusted according to the actual needs of the system. Rules can include conditional judgments on the type, data volume, source, etc. of the request to ensure the accuracy of the classification. Usually based on fixed conditions or thresholds, for example: when the amount of data requested exceeds a certain threshold, it is judged as a high-concurrency request.

[0158] When the request control module in the control layer is started, the system configures the monitored network port or open API interface. The monitoring service will continue to run so as to receive requests from various application components in real time. After receiving a request, the request control module will first parse the request. The parsing operation includes reading the basic information in the request header, such as the request type, data size, request source, etc., and parsing the specific operation parameters in the request body. The extraction of this information is the basis for subsequent classification. Based on the parsed request information, the module determines whether the request belongs to "normal request" or "high concurrency request" by calling the preset classification rules. The basis for classification can be the amount of data requested, the complexity of the operation, the priority of the source user, etc. According to the classification results, the request control module will select different processing paths, leaving the normal request in the control layer for processing, and forwarding the high concurrency request to the data layer for processing.

[0159] This embodiment can effectively select the optimal processing path according to the characteristics of the request by listening to and receiving requests in the control layer, and parsing and classifying the requests. The request control module can flexibly determine the request type and schedule it according to the preset classification rules, thereby improving the utilization of system resources and reducing the processing pressure in high-concurrency scenarios. At the same time, the module supports dynamic adjustment of classification rules to ensure adaptability and flexibility in different business scenarios.

[0160] In one embodiment, in the above S50, if the request is a common request, accessing the target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer includes:

[0161] S501, pre-configuring access interfaces through the request control module, and defining specific functions and applicable scenarios of each access interface;

[0162] S502, determining a target access interface based on the type of the common request and in combination with specific functions and applicable scenarios of each access interface;

[0163] S503: Acquire target information from the distributed cache module by calling the target access interface according to the control connection parameters.

[0164] In this embodiment, multiple standardized access interfaces are pre-configured in the request control module of the control layer. Each access interface is designed according to different functions and application scenarios, such as account balance query interface, transaction record query interface, etc. The configuration of the interface includes setting the function of each interface, supported request types, processing logic, etc.

[0165] After receiving a common request, the request control module dynamically selects an access interface that is most suitable for processing the common request based on the type of request (such as query type, write type, etc.) and the applicable scenario of each access interface. This selection is based on multiple dimensions such as the functional requirements of the request, the amount of data accessed, and the response time.

[0166] After the request control module selects the appropriate target access interface, it will establish a connection with the distributed cache module and obtain the target information by calling the target access interface according to the control connection parameters (such as connection pool size, timeout, number of retries, etc.). The control connection parameters play a role in stabilizing and efficiently managing the connection in this process.

[0167] When the request control module is started, multiple access interfaces are predefined and configured through the management platform or configuration file. The configuration of each access interface includes the function definition of the interface, the supported request types and the applicable scenarios. For example, an interface can be defined as an "account information query interface" suitable for the scenario of fast query of account data. After the request control module receives the general request and completes the preliminary analysis, the module will select the appropriate access interface according to the type of request (such as reading, query, etc.), combined with the specific functions and applicable scenarios of each interface. The module can complete this selection by querying the interface registry or consulting the preset interface mapping relationship. After the target interface is determined, the request control module will call the interface and manage the connection resources with reference to the control connection parameters. The control connection parameters include the size of the connection pool, the timeout period, etc. The module establishes a connection with the distributed cache module by calling the interface and performs data query or other operations according to the functional logic of the interface. By calling the target access interface, the request control module queries the target information from the distributed cache module. After successfully obtaining the information, the module returns the data to the application component that initiated the request according to the preset path.

[0168] This embodiment can significantly improve the system's processing efficiency for common requests by pre-configuring access interfaces and defining the functions and applicable scenarios of each interface. By dynamically selecting the appropriate target access interface, the system can flexibly adapt to different request types and scenarios and optimize the data access path. Combined with the management of control connection parameters, the call of the access interface can achieve fast data access and response while maintaining stability.

[0169] In one embodiment, the above S10 includes:

[0170] S101, deploying the data layer on a group of independent physical servers or virtual machines;

[0171] S102, deploying the control layer on another group of independent physical servers or virtual machines;

[0172] S103: Configure a network connection between the data layer and the control layer.

[0173] In this embodiment, the main function of the data layer is to process high-concurrency data access requests and perform operations related to data storage and caching. In order to ensure the efficient operation of the data layer, the data layer is deployed separately on an independent physical server or virtual machine. The purpose of this is to improve the specialization of computing resources and reduce the delay of data processing.

[0174] The control layer receives and parses requests from various application components, and schedules and processes them according to the request type. The control layer is deployed independently from the data layer and is mainly responsible for the classification and scheduling of requests. By deploying the control layer independently, the computing tasks of request scheduling and data processing can be effectively isolated, improving the scalability of the system.

[0175] Since the data layer and the control layer are deployed on different physical servers or virtual machines, an efficient network connection needs to be established to ensure the communication speed and reliability between the data layer and the control layer. The configuration of the network connection can include selecting the appropriate network protocol, bandwidth and latency optimization strategy, etc.

[0176] When the system is initialized, the service programs of the data layer are deployed on a group of physical servers or virtual machines. The deployment of the data layer can be completed through automated deployment tools (such as Ansible, Kubernetes, etc.) to ensure high availability and consistency of services. After the deployment is completed, the data layer will focus on processing high-concurrency requests from the control layer. When the system is initialized, the service programs of the control layer are deployed on another group of physical servers or virtual machines. The deployment of the control layer can also be completed through automated deployment tools. After the deployment is completed, the control layer will focus on receiving and parsing requests from various application components, and classifying and scheduling them. During the configuration process of the system, it is necessary to ensure that the network connection between the data layer and the control layer is unobstructed. High-speed network links and optimized network protocols (such as TCP / IP, gRPC, etc.) are usually used to ensure efficient data transmission. Connection timeout and retry policies can also be set in the network configuration to ensure reliable communication between the data layer and the control layer.

[0177] This embodiment can significantly improve the resource utilization and scalability of the system by dividing the data processing architecture into a data layer and a control layer. The data layer focuses on high-concurrency data processing, while the control layer is responsible for the classification and scheduling of requests. This layered architecture can effectively reduce the coupling of the system and improve the independence and flexibility of each layer. In addition, through independent physical servers or virtual machine deployment, the computing resources of the data layer and the control layer can be dynamically expanded according to actual business needs, thereby improving the processing capacity of the system.

[0178] In one embodiment, a data processing device based on a layered architecture is provided, and the data processing device based on a layered architecture corresponds one-to-one to the data processing method based on a layered architecture in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of a data processing device based on a layered architecture of the present invention. Architecture management module 10, concurrent data processing module 20, request control module 30, request identification module 40, common request processing module 50 and concurrent data processing module 60. The functional modules are described in detail as follows:

[0179] The architecture management module 10 is used to divide the data processing architecture into a data layer and a control layer;

[0180] A concurrent data processing module 20, used to deploy a concurrent data processing module in the data layer and configure concurrent connection parameters between the concurrent data processing module and the distributed cache module;

[0181] A request control module 30, used to deploy a request control module in the control layer and configure control connection parameters between the request control module and the distributed cache module;

[0182] A request identification module 40, configured to receive a request from an application component in the control layer and determine a type of the request, wherein the type of the request includes a common request or a high-concurrency request;

[0183] A common request processing module 50, for accessing target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer if the request is a common request, and returning the target information to the application component that initiated the request;

[0184] The concurrent data processing module 60 is used to send the high-concurrency request to the data layer if the request is a high-concurrency request, access the target information from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer, and return the target information to the application component that initiated the request.

[0185] In one embodiment, the concurrent data processing module 60 is specifically used to:

[0186] When the concurrent data processing module fails to successfully access the target information from the distributed cache module, acquiring the target information from the database through the concurrent data processing module;

[0187] Cache the target information in the distributed cache module;

[0188] The target information is returned to the application component that initiated the request.

[0189] In one embodiment, the concurrent data processing module 20 is specifically used for:

[0190] Installing the concurrent data processing module on a physical server or a virtual machine of the data layer;

[0191] In the concurrent data processing module, concurrent connection parameters with the distributed cache module are set, and the concurrent connection parameters include connection pool size, timeout time and number of retries;

[0192] The connection pool is initialized according to the connection pool size, and available connections with the distributed cache module are pre-allocated in the connection pool.

[0193] In one embodiment, the concurrent data processing module 60 is specifically used for:

[0194] Determine an idle connection from a pre-initialized connection pool through the concurrent data processing module, and perform a data access operation in the distributed cache module through the idle connection;

[0195] Monitoring the running status of the current connection through the concurrent data processing module, and judging whether the current connection has a connection timeout according to the timeout time in the concurrent connection parameters;

[0196] If the concurrent data processing module detects that a connection timeout occurs in the current connection, the current connection is released, and a connection with the distributed cache module is reestablished according to the number of retries in the concurrent connection parameter;

[0197] When the target information is successfully accessed, the concurrent data processing module releases and returns the currently used connection to the connection pool.

[0198] In one embodiment, the request identification module 40 is specifically configured to:

[0199] Monitoring and receiving requests from application components through the request control module in the control layer;

[0200] Parsing the request and extracting parameter information in the request header and the request body from the request;

[0201] According to the parameter information, the requests are classified into common requests or high-concurrency requests by using preset classification rules.

[0202] In one embodiment, the common request processing module 50 is specifically used for:

[0203] Pre-configure access interfaces through the request control module and define specific functions and applicable scenarios of each access interface;

[0204] Determine the target access interface based on the type of the common request and in combination with the specific functions and applicable scenarios of each access interface;

[0205] According to the control connection parameters, the target information is acquired from the distributed cache module by calling the target access interface.

[0206] In one embodiment, the architecture management module 10 is specifically configured to:

[0207] Deploy the data layer on a set of independent physical servers or virtual machines;

[0208] Deploy the control layer on another set of independent physical servers or virtual machines;

[0209] A network connection between the data layer and the control layer is configured.

[0210] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the service side of a data processing method based on a layered architecture.

[0211] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a data processing method based on a layered architecture

[0212] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the following steps are implemented:

[0213] Divide the data processing architecture into the data layer and the control layer;

[0214] In the data layer, deploy a concurrent data processing module, and configure concurrent connection parameters between the concurrent data processing module and the distributed cache module;

[0215] In the control layer, a request control module is deployed, and control connection parameters between the request control module and the distributed cache module are configured;

[0216] In the control layer, a request from an application component is received, and a type of the request is determined, where the type of the request includes a common request or a high-concurrency request;

[0217] If the request is a common request, access the target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer, and return the target information to the application component that initiated the request;

[0218] If the request is a high-concurrency request, the high-concurrency request is sent to the data layer, and the concurrent data processing module of the data layer accesses the target information from the distributed cache module according to the concurrent connection parameters, and returns the target information to the application component that initiated the request.

[0219] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0220] Divide the data processing architecture into the data layer and the control layer;

[0221] In the data layer, deploy a concurrent data processing module, and configure concurrent connection parameters between the concurrent data processing module and the distributed cache module;

[0222] In the control layer, a request control module is deployed, and control connection parameters between the request control module and the distributed cache module are configured;

[0223] In the control layer, a request from an application component is received, and a type of the request is determined, where the type of the request includes a common request or a high-concurrency request;

[0224] If the request is a common request, access the target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer, and return the target information to the application component that initiated the request;

[0225] If the request is a high-concurrency request, the high-concurrency request is sent to the data layer, and the concurrent data processing module of the data layer accesses the target information from the distributed cache module according to the concurrent connection parameters, and returns the target information to the application component that initiated the request.

[0226] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0227] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0228] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0229] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data processing method based on a layered architecture, characterized in that: The following steps are involved: Divide the data processing architecture into the data layer and the control layer; In the data layer, deploy a concurrent data processing module, and configure concurrent connection parameters between the concurrent data processing module and the distributed cache module; In the control layer, a request control module is deployed, and control connection parameters between the request control module and the distributed cache module are configured; In the control layer, a request from an application component is received, and a type of the request is determined, where the type of the request includes a common request or a high-concurrency request; If the request is a common request, access the target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer, and return the target information to the application component that initiated the request; If the request is a high-concurrency request, the high-concurrency request is sent to the data layer, and the concurrent data processing module of the data layer accesses the target information from the distributed cache module according to the concurrent connection parameters, and returns the target information to the application component that initiated the request.

2. The data processing method based on a layered architecture as claimed in claim 1, characterized in that: After accessing the target information from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer, the method further includes: When the concurrent data processing module fails to successfully access the target information from the distributed cache module, acquiring the target information from the database through the concurrent data processing module; Cache the target information in the distributed cache module; The target information is returned to the application component that initiated the request.

3. The data processing method based on layered architecture according to claim 1, characterized in that: In the data layer, a concurrent data processing module is deployed, and concurrent connection parameters between the concurrent data processing module and the distributed cache module are configured, including: Installing the concurrent data processing module on a physical server or a virtual machine of the data layer; In the concurrent data processing module, concurrent connection parameters with the distributed cache module are set, and the concurrent connection parameters include connection pool size, timeout time and number of retries; A connection pool is initialized according to the connection pool size, and available connections to the distributed cache module are pre-allocated in the connection pool.

4. The data processing method based on layered architecture according to claim 1, characterized in that: Accessing target information from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer includes: Determine an idle connection from a pre-initialized connection pool through the concurrent data processing module, and perform a data access operation in the distributed cache module through the idle connection; Monitoring the running status of the current connection through the concurrent data processing module, and judging whether the current connection has a connection timeout according to the timeout time in the concurrent connection parameter; If the concurrent data processing module detects that a connection timeout occurs in the current connection, the current connection is released, and a connection with the distributed cache module is reestablished according to the number of retries in the concurrent connection parameter; When the target information is successfully accessed, the concurrent data processing module releases and returns the currently used connection to the connection pool.

5. The data processing method based on a layered architecture as claimed in claim 1, characterized in that: In the control layer, a request from an application component is received and the type of the request is determined, where the type of the request includes a common request or a high-concurrency request, including: Monitoring and receiving requests from application components through the request control module in the control layer; Parsing the request and extracting parameter information in the request header and the request body from the request; According to the parameter information, the requests are classified into common requests or high-concurrency requests by using preset classification rules.

6. The data processing method based on layered architecture according to claim 1, characterized in that: If the request is a common request, accessing target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer includes: Pre-configure access interfaces through the request control module and define specific functions and applicable scenarios of each access interface; Determine the target access interface based on the type of the common request and in combination with the specific functions and applicable scenarios of each access interface; According to the control connection parameters, the target information is acquired from the distributed cache module by calling the target access interface.

7. The data processing method based on layered architecture according to claim 1, characterized in that: The data processing architecture is divided into the data layer and the control layer, including: Deploy the data layer on a set of independent physical servers or virtual machines; Deploy the control layer on another set of independent physical servers or virtual machines; A network connection between the data layer and the control layer is configured.

8. A data processing device based on a layered architecture, characterized in that: The data processing device based on the layered architecture comprises: An architecture management module, used to divide the data processing architecture into a data layer and a control layer; A concurrent data processing module, used to deploy the concurrent data processing module in the data layer and configure concurrent connection parameters between the concurrent data processing module and the distributed cache module; A request control module, used to deploy the request control module in the control layer and configure control connection parameters between the request control module and the distributed cache module; A request identification module, used to receive a request from an application component in the control layer and determine the type of the request, wherein the type of the request includes a common request or a high-concurrency request; A common request processing module, for accessing target information from the distributed cache module according to the control connection parameters through the access interface provided by the request control module of the control layer if the request is a common request, and returning the target information to the application component that initiated the request; A concurrent data processing module is used to send the high-concurrency request to the data layer if the request is a high-concurrency request, access the target information from the distributed cache module according to the concurrent connection parameters through the concurrent data processing module of the data layer, and return the target information to the application component that initiated the request.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a data processing program based on a layered architecture that is stored in the memory and can be run on the processor. When the data processing program based on a layered architecture is executed by the processor, the steps of the data processing method based on a layered architecture are implemented as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a data processing program based on a layered architecture, and when the data processing program based on a layered architecture is executed by a processor, the steps of the data processing method based on a layered architecture as described in any one of claims 1-7 are implemented.

Images