A data processing method, device, apparatus, and storage medium

Abstract

The application discloses a data processing method, device and equipment, and a storage medium. The method comprises the following steps: receiving a first request, and determining a data type of target data requested by the first request; determining a target region from at least one region included in a cache based on the data type of the target data; different regions in the at least one region are used for storing data of different data types; and performing processing corresponding to the first request on the target data in the target region. Through the scheme, the response time of data processing can be reduced, and the timeliness requirement of data processing can be met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and is related to but not limited to a data processing method, apparatus, device and storage medium. Background Art

[0002] With the development of internet technology, smart healthcare has gained widespread application, and various smart healthcare applications (APPs) have gradually entered people's lives. These apps need to process enormous amounts of data, placing high demands on timeliness and availability.

[0003] In related technologies, the process of processing large amounts of data may include: the first solution is to store the data in a unified database, and when processing the data, call the target data from the database in a traversal manner, and perform relevant processing on the target data; the second solution is to store the data in a cache through a middleware, and when processing the data, call the target data from the cache in a traversal manner through the middleware, and perform relevant processing on the target data.

[0004] For the above two solutions, when processing data, it is necessary to search for target data by traversing the huge storage space. This results in a long response time and low data processing efficiency, which cannot meet the timeliness requirements of data processing. Summary of the Invention

[0005] The present application provides a data processing method and apparatus, equipment, and storage medium that can reduce the response time of data processing, improve data processing efficiency, and meet the timeliness requirements of data processing.

[0006] The technical solution of this application is achieved as follows:

[0007] The present application provides a data processing method, the method comprising:

[0008] receiving a first request, and determining a data type of target data to be processed by the first request;

[0009] Determining a target area from at least one area included in the cache based on a data type of the target data; wherein different areas in the at least one area are used to store data of different data types;

[0010] The processing corresponding to the first request is performed on the target data in the target area.

[0011] The present application provides a data processing device, comprising:

[0012] a receiving unit, configured to receive a first request and determine a data type of target data to be processed by the first request;

[0013] a determining unit, configured to determine a target area from at least one area included in the cache based on a data type of the target data; different areas in the at least one area are used to store data of different data types;

[0014] A processing unit is configured to execute processing corresponding to the first request on the target data in the target area.

[0015] The present application also provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor implements the above-mentioned data processing method when executing the program.

[0016] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data processing method is implemented.

[0017] The data processing method, apparatus, device and storage medium provided by the present application include: receiving a first request, and determining the data type of target data to be processed by the first request; determining a target area from at least one area included in a cache based on the data type of the target data; different areas in the at least one area are used to store data of different data types; and performing processing corresponding to the first request on the target data in the target area. The present application provides a data processing solution that stores data of different data types in different areas of the cache. In this way, when performing data processing, the data type of the target data is first determined based on the request, and then the target area is determined based on the data type. Then, data processing is performed on the target data in the target area, thereby reducing the response time of data processing, improving data processing efficiency, and meeting the timeliness requirements of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 An optional structural diagram of a data processing system provided in an embodiment of the present application;

[0019] Figure 2 An optional flowchart of the data processing method provided in the embodiment of the present application;

[0020] Figure 3 An optional flowchart of the data processing method provided in the embodiment of the present application;

[0021] Figure 4 An optional flowchart of the data processing method provided in the embodiment of the present application;

[0022] Figure 5 An optional structural diagram of a data processing system provided in an embodiment of the present application;

[0023] Figure 6 An optional structural diagram of the data synchronization process provided in an embodiment of the present application;

[0024] Figure 7 An optional structural diagram of an order push method provided for related technologies;

[0025] Figure 8 An optional structural diagram of the pre-calculation process provided in an embodiment of the present application;

[0026] Figure 9 A schematic diagram of an optional structure of the prescription system and order cache provided in an embodiment of the present application;

[0027] Figure 10 A schematic diagram of an optional structure of the prescription system and order cache provided in an embodiment of the present application;

[0028] Figure 11 An optional structural diagram of the content area and relationship area provided in an embodiment of the present application;

[0029] Figure 12 A schematic diagram of an optional structure of a data processing device provided in an embodiment of the present application;

[0030] Figure 13 This is a schematic diagram of an optional structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the specific technical solutions of the application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0032] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first, second, and third" are used merely as examples to distinguish between different objects and do not represent a specific order or precedence for the objects. It is understood that the specific order or precedence of "first, second, and third" can be interchanged where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0035] The embodiments of the present application may provide a data processing method and apparatus, a device, and a storage medium. In practical applications, the data processing method may be implemented by a data processing apparatus, and the functional entities in the data processing apparatus may be collaboratively implemented by hardware resources of an electronic device (e.g., a terminal device), such as computing resources such as a processor, and communication resources (e.g., for supporting various communication methods such as optical cables and cellular communications).

[0036] The data processing method provided in the embodiment of the present application is applied to a data processing system, and the data processing system includes a data processing end.

[0037] A data processing end receives a first request and determines the data type of target data to be processed by the first request; determines a target area from at least one area included in a cache based on the data type of the target data; different areas in the at least one area are used to store data of different data types; and performs processing corresponding to the first request on the target data in the target area.

[0038] The data processing system may also include clients.

[0039] As an example, the structure of the data processing system may be as follows Figure 1 As shown, it includes: a client 10 and a data processing terminal 20.

[0040] In one example, the client 10 and the data processing end 20 may be the same physical entity; in another example, Figure 1 As shown, the client 10 and the data processing end 20 may be different physical entities, and the client 10 and the data processing end 20 interact with each other through a network 30 .

[0041] Here, the client 10 is used to receive a user operation, and based on the user operation, send a first request to the data processing end 20. The data processing end 20 receives the first request sent by the client 10.

[0042] Next, combine Figure 1 The schematic diagram of the data processing device shown in the figure illustrates various embodiments of the data processing method and device, equipment and storage medium provided in the embodiments of the present application.

[0043] An embodiment of the present application provides a data processing method, which is applied to a data processing device, wherein the data processing device can be deployed on an electronic device serving as a data processing terminal.

[0044] Figure 2A flow chart illustrating an optional data processing method is provided. The data processing method provided in the embodiment of the present application is used to process data. Taking target data as an example, the data process of the target data is now described in detail.

[0045] The data processing method may include but is not limited to Figure 2 The following S201 to S203 are shown.

[0046] S201: The electronic device receives a first request and determines the data type of target data to be processed by the first request.

[0047] The embodiment of the present application does not limit the operation type of the first request and can be configured according to actual needs. The operation type of the first request may include but is not limited to at least one of the following: read request, write request, query request, delete request, modify request, and update request.

[0048] In a possible implementation, the electronic device receives a first operation from a user, receives a first request based on the first operation, and determines a data type of target data to be processed by the first request.

[0049] In another possible implementation, the client receives a first operation from the user and receives a first request based on the first operation. The client sends the received first request to the electronic device. The electronic device receives the first request sent by the client and determines the data type of the target data to be processed by the first request.

[0050] It should be noted that the client sending the received first request to the electronic device may include: the client sending the received first request directly to the electronic device, that is, the first message is sent without passing through any device other than the electronic device and the client; or, the client sending the received first request to the first device (a device other than the electronic device and the client), and the first device forwarding the first request to the electronic device.

[0051] The embodiment of the present application does not specifically limit the data type and can be configured according to actual needs. For example, the data type may include at least one of the following: order data, static data, status data, and count data.

[0052] For example, when the first request received is “read the number of orders today”, the electronic device determines that the target data is counting data.

[0053] S202: The electronic device determines a target area from at least one area included in the cache based on a data type of the target data.

[0054] In the at least one region, different regions are used to store data of different data types.

[0055] The embodiment of the present application does not limit the number of areas included in the cache, nor the different types of data stored in different areas, and can be configured according to actual needs.

[0056] Example A1, the cache may include 4 areas, namely the first area, the second area, the third area, and the fourth area, wherein the first area is used to store order data, the second area is used to store static data, the third area is used to store status data, and the fourth area is used to store count data.

[0057] Example A2, the cache may include 6 areas, namely the first area, the second area, the third area, the fourth area, the fifth area, and the sixth area, wherein the first area is used to store received order data, the second area is used to store order data output by the user (such as a doctor), the third area is used to store paragraph-like static data, the fourth area is used to store noun-like static data, the fifth area is used to store status data, and the sixth area is used to store count data.

[0058] S202 may be implemented as follows: the electronic device determines, based on the data type of the target data, an area for storing the target data from at least one area included in the cache as the target area.

[0059] For example A1, S202 can be example B1: if the electronic device determines that the data type of the target data is an order data type, the target area is determined to be the first area; if the electronic device determines that the data type of the target data is a static data type, the target area is determined to be the second area; if the electronic device determines that the data type of the target data is a status data type, the target area is determined to be the third area; if the electronic device determines that the data type of the target data is a counting data type, the target area is determined to be the fourth area.

[0060] S203: The electronic device performs processing corresponding to the first request on the target data in the target area.

[0061] The electronic device performs a process corresponding to the first request on the target data in the target area.

[0062] In a possible implementation, the first request is a read request, and the electronic device reads target data in the target area.

[0063] In a possible implementation, the first request is a write request, and the electronic device writes target data into a target area.

[0064] The present embodiment does not limit the order in which the target data is written, and can be configured according to actual needs. For example, the electronic device can write the target data in the order of writing time; or the electronic device can write the target data in a preset order (for example, the order of order completion time, the preset department order, etc.).

[0065] The data processing method provided by the embodiment of the present application includes: receiving a first request, and determining the data type of target data to be processed by the first request; determining a target area from at least one area included in the cache based on the data type of the target data; different areas in the at least one area are used to store data of different data types; and performing processing corresponding to the first request on the target data in the target area. The data processing method provided by the present application stores data of different data types in different areas of the cache. In this way, when performing data processing, the data type of the target data is first determined based on the request, and then the target area is determined based on the data type. Then, data processing is performed on the target data in the target area, thereby reducing the response time of data processing, improving data processing efficiency, and meeting the timeliness requirements of data processing.

[0066] The execution of S201 and S203 is briefly described below, which may include but is not limited to the following method 1 or method 2.

[0067] Method 1: Execute S201 and S203 through the same task;

[0068] Method 2: Execute S201 and S203 through different tasks.

[0069] Mode 1: The electronic device receives a first request through a first task and determines the data type of target data to be processed by the first request; the electronic device performs processing corresponding to the first request on the target data in a target area through the first task.

[0070] Method 1 can implement the actions of receiving the first request and processing the first request through one task, which is simple to implement and can further reduce the response time of data processing.

[0071] Mode 2: The electronic device receives the first request through the first task; the electronic device performs processing corresponding to the first request on the target data in the target area through the second task.

[0072] The first task is different from the second task. The embodiment of the present application does not limit the task type and execution timing of the first task and the second task, and can be configured according to actual needs. For example, with respect to the task type, the first task can be a foreground task, and the second task can be a background task; with respect to the execution actual, the first task can be a real-time processing task, and the second task can be a real-time processing task or an idle processing task.

[0073] Method 2 allows the first task to receive the first request, determine the data type of the target data to be processed by the first request, and then process the first request through the second task. This allows for flexible configuration of the first and second tasks based on actual needs. Furthermore, if the first task is a foreground task and the second task is a background task, the user is unaware of the processing of the first request, improving the user experience.

[0074] It should be noted that the present embodiment does not limit the task of executing S202 and can be configured according to actual needs. For example, the task of executing S202 can be the same as the task of executing S201; or the same as the task of executing S203; or S202 can be executed by a third task different from the first and second tasks.

[0075] Next, the process of the electronic device executing the processing corresponding to the first request on the target data in the target area at S203 is described, which may specifically include but is not limited to any of the following methods A1 to A4:

[0076] Method A1 in the scenario where the target area is the first area, where the first area is used to store order data;

[0077] Method A2 in a scenario where the target area is the second area, where the second area is used to store static data;

[0078] Method A3 in the scenario where the target area is the third area, where the third area is used to store status data;

[0079] In the scenario where the target area is the fourth area, in mode A4, the fourth area is used to store counting data.

[0080] In the following, the implementation process of mode A1, in which the target area is the first area and the electronic device performs the processing corresponding to the first request on the target data in the target area, is described.

[0081] Method A1 may include but is not limited to Figure 3 S301 and S302 shown.

[0082] S301: The electronic device determines a first target sub-area in the first area.

[0083] Different sub-areas in the first area are used to store order data of different users. For example, different sub-areas in the first area are used to store order data of different doctors. The order data may include: received orders and output orders.

[0084] In a possible implementation, when the first request is a write request, the electronic device determines a first target sub-region in the first region according to a preset rule.

[0085] The preset rule is used to determine the first target sub-area in the first area. The embodiment of the present application does not limit the specific content of the preset rule, and it can be configured according to actual needs.

[0086] The preset rule may include: determining the first target sub-area according to the time when the request is received.

[0087] For example, if the first request is for writing order 1 and the first request is received at 10:00, the first target sub-region is the sub-region storing the order data of doctor A. This can also be understood as assigning the first request to doctor A for processing.

[0088] The preset rule may further include: determining the first target sub-area based on a second operation of the user in response to the first request.

[0089] For example, if the first request is for writing order 2, and user B (doctor B) performs a second operation (also called an order grabbing operation) in response to the first request, the electronic device determines, based on the second operation, that the first target sub-region is the sub-region storing user B's order data. This can also be understood as assigning the first request to doctor B for processing.

[0090] In a possible implementation, when the first request is a read request, the electronic device determines a sub-region in the first region that is associated with the first request as the first target sub-region.

[0091] For example, the first request is to read order 3, where order 3 of the first request is associated with the sub-area storing the order data of user C (order 3 belongs to doctor C), then the electronic device determines the sub-area storing the order data of user C as the first target sub-area.

[0092] S302: The electronic device performs processing corresponding to the first request on the target data in the first target sub-area.

[0093] In a possible implementation, the first request is a read request, and the electronic device reads target data in the first target sub-region.

[0094] In a possible implementation, the first request is a write request, and the electronic device writes target data into the first target sub-region.

[0095] Next, the implementation process of the method A2 is described in which the target area is the second area and the electronic device performs the processing corresponding to the first request on the target data in the target area. Method A2 may include but is not limited to Figure 4 S401 to S403 shown.

[0096] S401: The electronic device determines the access status of the target data.

[0097] The electronic device determines whether the access frequency of the target data is greater than or equal to a first frequency threshold. If the access frequency of the target data is greater than or equal to the first frequency threshold, the access state of the target data is determined to be the first state; if the access frequency of the target data is less than the first frequency threshold, the access state of the target data is determined to be the second state.

[0098] S402: The electronic device determines a second target sub-area in the second area based on the access status.

[0099] Different sub-areas in the second area are used to store static data in different access states.

[0100] The electronic device determines, in the second area, a sub-area storing the access status data of the target data as a second target sub-area.

[0101] S403: The electronic device performs processing corresponding to the first request on the target data in the second target sub-area.

[0102] In a possible implementation, the first request is a read request, and the electronic device reads target data in the second target sub-region.

[0103] In a possible implementation, the first request is a write request, and the electronic device writes the target data into the second target sub-area.

[0104] Next, a process of determining the second target sub-area in the second area based on the access status at S402 by the electronic device will be described, which may include but is not limited to the following implementation A or implementation B.

[0105] Implementation A: for the first state, determining the first sub-region as the second target sub-region;

[0106] Implementation method B: for the second state, determining the second sub-region as the second target sub-region.

[0107] Implementation A: If the access state of the target data is the first state representing hotspot data, the electronic device determines the first sub-area as the second target sub-area, wherein the first sub-area caches data through the first middleware.

[0108] Implementation method B: If the access state of the target data is the second state characterized as non-hotspot data, the electronic device determines the second sub-area as the second target sub-area, wherein the second sub-area caches data through the second middleware.

[0109] The data processing rate of the second middleware is lower than that of the first middleware. The embodiments of the present application do not limit the specific types of the first middleware and the second middleware, and they can be configured according to actual needs. For example, the first middleware can be a distributed cache (Memcached); the second middleware can be a remote dictionary server (RemoteDictionary Server, Redis).

[0110] Next, the implementation process of mode A3 in which the target area is the third area and the electronic device performs the processing corresponding to the first request on the target data in the target area is described.

[0111] The third area is used to store status data, where status data refers to data used to characterize the status. The embodiment of the present application does not specifically limit the status data and can be configured according to actual needs. For example, the status data may include: whether the message is read, whether the prescription is submitted, etc.

[0112] Mode A3: Execute the processing corresponding to the first request on the target data in the third region through a Bloom filter.

[0113] In one possible implementation, the first request is a write request, and the electronic device determines a mapping location corresponding to the write request in the third area using a Bloom filter. The electronic device converts the target data into a first value and then writes the first value into the mapping area. The first value is used to represent the target data.

[0114] In a possible implementation, the first request is a read request. The electronic device determines a mapping position corresponding to the read request in the third area through a Bloom filter, reads a value of the mapping position, and obtains target data.

[0115] Method 3 performs the processing corresponding to the first request on the target data in the target area. When caching state data, the state data is not cached directly. Instead, a Bloom filter is used to obtain the mapping location of the state data in the third area. The state data is converted into a first value (representing the first state) or a second value (representing the second state), and stored in the mapping location. This reduces storage capacity and improves storage efficiency, while also improving read efficiency.

[0116] Next, the implementation process of mode A4 in which the target area is the fourth area and the electronic device performs the processing corresponding to the first request on the target data in the target area is described.

[0117] The fourth area is used to store count data, where count data refers to data that can be counted. This embodiment of the application does not specifically limit count data and can be configured according to actual needs. For example, count data may include: historical order quantity, today's order quantity, historical patient quantity, number of departments, number of doctors, etc.

[0118] The fourth area is a cache area. The embodiment of the present application does not specifically limit the cache type of the fourth area, and can be configured according to actual needs. For example, the fourth area can be a high-speed cache memory (Cache).

[0119] In a possible implementation, the first request is a write request, and the electronic device determines that the target data is greater than or equal to 1, and writes the target data into the fourth area.

[0120] It should be noted that if the target data is equal to 0, the target data is not stored. In other words, the counting data with a count number of 0 is not stored.

[0121] In a possible implementation, the first request is a read request, and the electronic device reads target data in the fourth area.

[0122] It should be noted that, if all objects in the fourth area are traversed and no target data is found, the count corresponding to the target data is determined to be 0.

[0123] By using method 4, the processing corresponding to the first request is performed on the target data in the target area. On the one hand, the count data is stored in the cache, and the response time of data processing is short. On the other hand, the fourth area only stores data with a count greater than or equal to 1, and does not store data with a count equal to 0, thereby improving the utilization of the cache space.

[0124] Below, the data processing method provided in the embodiment of the present application is explained through specific application scenarios.

[0125] One of the most important functions in the doctor app of an Internet hospital is the order list service. This service is one of the functions most frequently used by doctors every day. It has the characteristics of high concurrency (large number of visits), complex functional logic, high effectiveness requirements, high availability requirements, and low fault tolerance. Due to the above characteristics of the order list service, it is particularly important to design a highly available, easily scalable, and high-performance order list service.

[0126] The following is an introduction to the above features:

[0127] 1. High traffic: The Doctor App has 100,000 registered doctors and 20,000 active doctors per day. Each doctor needs to use the order list service to provide consultation services to patients. The average number of queries per minute (Q / M) reaches 10,000. During the 618 and Double 11 shopping festivals, the number of queries per minute soared to 100,000, which is 10 times the normal number.

[0128] 2. Complex functional logic: The order list service is a high-frequency function used by doctors. This service integrates a rich set of business functions. Doctors can see the orders directed to them by patients (directed orders) and the orders assigned to them by the system based on the patient's description and information (dispatched orders). Similarly, doctors can grab orders they believe they can treat (grab orders). Ultimately, the computing, storage, synchronization, caching, and other functions evolved in the system architecture, which are extremely complex.

[0129] Static information category (equivalent to static data): order type, disease complaint, disease pictures uploaded by the patient, patient information (age, gender, name, etc.).

[0130] Dynamic calculation category: the content of the last sentence in the doctor-patient conversation, the countdown to the end of the order, whether there are unread messages that require a red dot reminder, which consultation buttons can appear (rejection, acceptance, referral, withdrawal, and termination), etc.

[0131] Existence category (equivalent to status data): whether the order has been responded to by the responsible doctor, whether it has been referred, whether it belongs to VIP, whether a prescription has been issued, whether the consultation can be postponed, etc.

[0132] Counting type (equivalent to counting data): how many people in the expert team have responded to the order, the number of historical consultations for the patient, etc.

[0133] Sorting rules: sort by order time, sort by the time when the patient replied to the last sentence, sort by order amount, etc.

[0134] 3. High effectiveness requirements: Due to the particularity of the medical industry, medical consultations and treatments are always urgent matters, so the order list service has very high effectiveness requirements. For example, after a prescription is issued, the prescription information must be synchronized to everyone in the expert team in real time, and some doctors must not prescribe the same prescription again; after the order time expires, the order status is automatically refreshed; doctors using this service must ensure smooth functionality, data acquisition must be within 5ms, and the response time must be within 10-40ms.

[0135] 4. High availability requirements: The core function of online medical care is for doctors and patients to communicate through the Internet through text, pictures, voice, video, etc., and the order list is the top priority for doctors to manage patients and provide services. If this service is unavailable, doctors and patients cannot communicate, and the entire Internet hospital is completely paralyzed. Therefore, availability must be above 99.99%.

[0136] 5. Low fault tolerance: If the service encounters problems such as lost orders, untimely messages, premature order termination, disordered session ordering, and untimely red dot prompts, it will cause inconvenience to doctors and ultimately affect the patient's consultation results, recovery progress, and other issues. Therefore, the service has a very low fault tolerance rate, which places high demands on the architectural design.

[0137] In related technologies, the process of processing data in the order list service includes the following technology 1 or technology 2:

[0138] Technique 1: Synchronous Calls: Retrieve relevant orders through a database (DB), performing online synchronous calls and temporarily calculating relevant data each time. Technique 1 uses a database as a storage medium. Since most databases use mechanical disks, their performance is not as efficient as caches. They cannot cope with the high pressure in high-traffic and high-concurrency scenarios. Especially during major promotions, the database is very likely to crash, causing serious online incidents.

[0139] Technique 2: Single Cache: Leveraging the high performance of Redis middleware, data is cached in a single Redis middleware. Data is retrieved from Redis each time and the final result is assembled and sent to the front-end. Technique 2 uses only a single Redis cache middleware, which solves most of the performance issues compared to the first solution. However, because Redis data capacity (single instance) is less than 30GB and content utilization is low, it is difficult to cover all scenarios and results in a significant waste of servers.

[0140] This application provides a data processing method that can provide users with a highly available, high-performance order list service, a data processing solution with correct functional logic and fast response.

[0141] First, a brief description of various cache middleware is given. Table 1 is a comparison example of cache middleware.

[0142] Table 1 Comparison examples of cache middleware

[0143]

[0144] Among them, through the comparison of various cache middlewares in Table 1, it is possible to flexibly configure suitable middleware for various data types to cache data, thereby improving processing efficiency.

[0145] A brief explanation of some technical terms involved in the embodiments of this application is given below.

[0146] Pre-computation: Store data in advance.

[0147] Message Queue (MQ): An internal message component that provides topic-level production and consumption functions.

[0148] Remote Dictionary Server (Redis): A multi-faceted in-memory storage middleware that provides in-memory performance.

[0149] Heterogeneous caching: Backing up data in different formats.

[0150] It is difficult for all of the above methods to solve the problem comprehensively. Therefore, this solution proposes a multi-set, multi-level, or multi-dimensional cache model solution with the following characteristics: pre-computation, asynchronous processing, heterogeneous caching, efficient response, unified platform, and data visualization.

[0151] The data processing method provided in the embodiment of the present application is applied to a data processing system. Figure 5 The structure of a data processing system is shown schematically.

[0152] like Figure 5 As shown, the processing system includes: a client device 501, a cache gateway 502, a configuration device 503, a cache proxy device 504, an update service policy device 505 and a storage device 506 (equivalent to the above electronic device or the above data processing end).

[0153] The client device 501 is configured to receive a first request based on a client operation and send the first request to the cache gateway 502 .

[0154] The cache gateway 502 is configured to receive the first request and distribute the first request to the gateway server closest to the user based on the user's location. A reverse proxy server (Nginx) is installed on the gateway server. This reverse proxy server utilizes a multiplexing concurrency mechanism (Epoll) model to easily address the c10k problem, which limits the number of connections a single server can handle to 10,000. Furthermore, Nginx, combined with Lua scripts, identifies the user's personal identification number (PIN) and intercepts repeated requests for the same PIN within a short period of time.

[0155] In the configuration of device 503, this application designs multi-dimensional redundant storage. Because the user base is large, different users access different caches, which can be diverted to reduce the pressure on the backend server. During use, different namespaces are registered for different users, and the corresponding cache middleware is found based on the namespace. This allows the backend cache to be added and removed, and the Internet Protocol (IP) to be changed, without the frontend being aware of the specific server changes.

[0156] Cache proxy device 504 is used to shield protocol differences between different caches. Because this application integrates multiple types of cache middleware, one function of cache proxy device 504 is to shield protocol differences between various cache middleware. Another function is read-write separation, that is, read operations access storage device 506, while write operations are diverted to update service policy device 505, where update service policy device 505 supports cache update policy service (update-server, us).

[0157] Update service policy device 505 is used to synchronize data to multiple servers efficiently, thereby comprehensively improving the user experience. Figure 6 As shown, data synchronization can include: cache proxy device 504 first analyzes whether the request instruction is read or write. If it is a read request, it directly reads the local cache. If it is a write request, it is first routed to update service policy device 505. Update service policy device 505 first updates the local resources and, if successful, routes it to the storage device 506 in the main computer room. The storage device 506 in the main computer room receives the write request and synchronizes it to the storage devices 506 in other slave computer rooms step by step, thereby achieving data synchronization.

[0158] The storage device 506 may include a receive queue area 5061, an output queue area 5062, a content area 5063, a relationship area 5064, a status area 5065, and a counter area 5066. The receive queue area 5061 and the output queue area 5062 correspond to the first area, the content area 5063 and the relationship area 5064 correspond to the second area, the status area 5065 corresponds to the third area, and the counter area 5066 corresponds to the fourth area.

[0159] Receive queue area 5061, output queue area 5062:

[0160] In related technologies, push mode is usually adopted, such as Figure 7 As shown, the patient places an orthopedic order 1, and this orthopedic order 1 is pushed to all orthopedic surgeons (orthopedic surgeon 1, orthopedic surgeon 2, etc.). The architecture is simple and easy to understand, but the disadvantage is that the performance is relatively low under high traffic.

[0161] There are 100,000 registered doctors online. Orthopedics, andrology, and internal medicine are all large departments with 10,000 to 20,000 registered doctors. Therefore, when a patient places an order, it is delivered to 10,000 doctors. This puts too much pressure on the server performance and is not advisable. Based on the above reasons, this solution designs a push-pull collection (hybrid) approach that includes pre-computing, data aggregation, event-driven and offline processing.

[0162] Pre-calculation

[0163] When a doctor (doctor D) requests access, the queue ( Figure 8 The data in the doctor D's queue is pre-calculated and stored, and is not calculated after receiving the doctor's access request. This speeds up the response speed and can reach a response within 40 milliseconds (ms).

[0164] Data Aggregation

[0165] A doctor's stored data sources may include one or more of the following:

[0166] 1) Orders placed by different patients;

[0167] When an order is related to a doctor, the doctor can accept the order and assign the orders in the output queue of different patients to the corresponding doctor's order list through event-driven processing.

[0168] 2) Orders in group chats;

[0169] Group chats generally include two categories: responsible housekeepers and expert teams. Doctors in the same team can obtain orders in the group chat.

[0170] 3) Grab the order;

[0171] Doctors grab orders according to their own wishes, and once the order is grabbed, it belongs to the doctor.

[0172] 4) Directed orders;

[0173] This type of order is placed by the patient to a certain doctor, usually a famous doctor from a tertiary hospital, and this type of order will be placed in the name of a certain doctor first.

[0174] 5) Message;

[0175] The messages mainly include doctor-patient conversations, usually the text of the last sentence and timestamp.

[0176] 6) Prescription information.

[0177] Prescription information includes the prescribed diagnosis, drug information, prescription status and other information.

[0178] like Figure 8 As shown, 1) (orders placed by patients) is the output queue, 2) to 6) (orders in group chats, grab orders, directed orders, messages, and prescription information) are all input queues. Different data is placed in different queues. Finally, the system aggregates the data and caches the queue storage areas of different doctors (such as the order list of doctor D).

[0179] Event-driven

[0180] Messages are driven by sending message events. For example, orders are divided into directed and non-directed orders. Directed orders are entered into the corresponding data queue through the order system, while non-directed orders are entered into the data queue through the triage system. The message system sends an event for every conversation between the doctor and the patient, continuously overwriting the data at the end of the data queue. The prescription system sends message events at multiple levels, including prescription issuance, submission, review, rejection, approval, and drug purchase. Message events avoid the unnecessary performance consumption caused by constant polling and are more timely.

[0181] Offline processing

[0182] When processing order data, we use offline processing instead of conventional synchronous processing. This is because synchronous processing puts enormous pressure on performance, and offline processing can effectively avoid this pressure. However, the problem brought about by this is that if there is a message backlog, the data will not be timely, and the doctor side will not be able to discover the patient's actions in time. Therefore, when dealing with such situations, we use semi-synchronous + message queue (MQ) compensation method:

[0183] like Figure 9 As shown, taking the relationship between the prescription system and the order cache as an example, if the prescription system adds data to the order cache side synchronously, it will seriously affect the prescription system. Therefore, MQ message events are preferably processed offline. However, if MQ backlog occurs, it will affect the entire system. Therefore, a layer of semi-synchronization is added in addition to the MQ message event, that is, an asynchronous task is submitted to the thread pool. The task is to synchronize data to the order cache. However, since the thread pool may also have backlogs, the risk is much smaller than the overall MQ backlog. Therefore, distributed services are currently used. If there is a problem with one service, other servers may not necessarily have problems, thereby achieving risk isolation.

[0184] Content area 5063, relationship area 5064:

[0185] The order list service needs to display the patient's name, gender, height, weight, past cases and other information. The entire Internet hospital has more than 60 million patient information in total. The doctor's name, department, age and other information are also needed. There are currently 100,000 doctors. There is a large amount of hot data in these static data. For example, a famous doctor receives more than a thousand orders every day. In addition, there is information about team members and team leaders. In summary, there is a large amount of basic data and a large amount of hot data. The data must be cached, and after caching, the data can be reused to provide multiple functions such as the doctor's homepage, search, and evaluation to reduce storage costs. Specifically, the embodiment of the present application adopts a four-layer architecture for the content area 5063 and the relationship area 5064 for storing static data. Figure 10 As shown, the L1 layer (equivalent to the first sub-area of ​​the second area) is used to store hotspot data, the L2 layer (equivalent to the second sub-area of ​​the second area) is used to store non-hotspot data, the L3 layer is used to back up non-hotspot data, and the L4 layer is used to back up all static data.

[0186] The L1 layer is used to store hot data, which may include data on popular doctors, popular hospitals, popular departments, popular teams, high-priced orders, the number of orders currently available in a department, and other data. Hot data is characterized by small storage capacity and fast response speed. First, the storage capacity is small, and it is stored in small blocks. The blocks are isolated from each other, which can prevent sudden traffic shocks that cause large-scale functional failures and can effectively isolate risks. Secondly, the response speed must be fast, so the L1 layer uses Memcached middleware for caching, which has a faster processing speed.

[0187] It should be noted that the hot data stored in the L1 layer is dynamically updated. For example, if some data is no longer popular, it will be downgraded to the L2 layer and no longer be hot data.

[0188] The L2 and L3 layers back up each other and store non-hotspot static data.

[0189] The non-hotspot data stored in the L2 and L3 layers are also dynamically updated. For example, when the access frequency of data 1 is greater than or equal to the first frequency threshold, data 1 will first be deleted from the L2 and L3 layers, and data 1 will be stored as hotspot data in the L1 layer.

[0190] The L4 layer is the DB layer, which is used to back up all static data.

[0191] Online verification and statistics show that the cache hit probability at the L1 layer is 64%, at the L2 and L3 layers is 34%, and at the database layer is 2%. Most cache hits are covered, significantly reducing database performance pressure. The L1 layer can be dynamically adjusted based on online traffic pressure. This means that if traffic increases, the L1 layer can be dynamically increased, and if traffic decreases, the L1 layer can be dynamically reduced.

[0192] In terms of storage media, the L1 layer uses Memcached, a high-performance binary simple KV storage. This is because hot data is often not very complex, so a simple KV model is sufficient. For example, the number of popular doctors is capped and cannot continue to increase. The L2 and L3 layers use Redis, a storage medium with a complex storage structure. This allows for information such as team relationships, using a hash structure to store team member information.

[0193] Status area 5065: The Internet Hospital Order List Service requires numerous validation tasks, such as whether a prescription has been issued, whether the recipient's message has been read, and whether the diagnosis has been submitted. Using Redis as a storage medium would be prohibitive due to the large storage requirements. Therefore, the solution in this application utilizes a Bloom filter, significantly reducing storage requirements.

[0194] For example, the process of using a Bloom filter to determine whether a message has been read may include: Figure 11 As shown, the message identifier is mapped to three positions (the mapping positions of message 1) using three hash functions. If the values ​​of all three positions are 1, message 1 is considered to have been read. If the value of one position is 0, message 1 is considered to have not been read. For example, if the identifier of message 1 is mapped to three positions (the mapping positions of message 1) using three hash functions, and the values ​​of all three positions are 1, message 1 is considered to have been read.

[0195] Counting area 5066 is used to store counting data, which may include: historical order volume, today's order volume, order volume in the last two hours, the total number of patients served so far, the number of teams currently joined, etc. A large number of counting tasks exist in the order list service. This solution provides two solutions here: one is DB+Cache, and the other is pure Cache. The first solution uses historical order volume as an example, storing past order volume in the DB on a daily basis, and dynamically calculating today's cache. If you want to view today's order volume, you can use the cache. If you want to view the total historical order volume, you can add the sum of the data in the DB to today's data volume. This solution is suitable for small data volumes, but once data skew occurs, for example, if the historical order volume of a famous doctor is very complex, reading counting data from the DB multiple times will also consume database performance and may even cause disasters. The pure cache method utilizes Redis's auto-increment (INCR) and auto-decrement (DECR) instructions. Data with a count greater than 0 is stored in the database, while data with a count of 0 does not need to be stored. This saves a lot of memory space. To prevent cache crashes, data must be stored in the database first and then in the cache to prevent data confusion and loss. This ensures a balance between security and performance.

[0196] An embodiment of the present application provides a toolkit that can automatically report log points when operating the cache, including basic information such as log identifier, timestamp, data source, data snapshot, etc., and use it through internal high-speed JDQ.

[0197] The interface monitoring function of the embodiment of the present application displays core business data through the interface based on log reports, which is used to monitor health procedures and system risks, and automatically notify operation and maintenance personnel if there are any problems.

[0198] The data processing method provided in the embodiment of the present application has the characteristics of pre-computation, asynchronous processing, heterogeneous caching, efficient response, unified platform and data visualization through a comprehensive solution of pre-computation, efficient response and data visualization.

[0199] Pre-calculation, asynchronous processing

[0200] Most operations in this solution are MQ asynchronous. The pre-calculation stores the data and waits for the user to retrieve it. When the user performs the operation, there is actually no calculation logic, only reading operations. This method can ensure that each response is less than 40ms.

[0201] Heterogeneous caching and efficient response

[0202] All data in this solution is backed up in DB or ES. The data structures are very different. To ensure efficient response, the data is flattened and heterogeneously stored in the cache medium in the form of KV, thereby achieving efficient response.

[0203] Unified platform, data visualization

[0204] In order to better monitor the health of the system, the entire system process is covered, buried data is reported, the data is visible, corrective actions are enabled, problems are discovered and corrected in a timely manner, and risks are avoided.

[0205] In order to implement the data processing method provided in the embodiment of the present application, a data processing device according to the embodiment of the present application is provided below in combination with Figure 12 The structural diagram of the data processing device is shown in FIG.

[0206] like Figure 12 As shown, the data processing device 120 includes: a receiving unit 1201, a determining unit 1202 and a processing unit 1203.

[0207] The receiving unit 1201 is configured to receive a first request and determine a data type of target data to be processed by the first request;

[0208] A determining unit 1202 is configured to determine a target region from at least one region included in the cache based on a data type of the target data; different regions in the at least one region are used to store data of different data types;

[0209] The processing unit 1203 is configured to execute processing corresponding to the first request on the target data in the target area.

[0210] In some embodiments, the receiving unit 1201 is further configured to receive the first request through the first task, and determine a data type of target data to be processed by the first request;

[0211] The processing unit 1203 is further configured to execute processing corresponding to the first request on the target data in the target area through a second task; the first task is different from the second task.

[0212] In some embodiments, the processing unit 1203 is further configured to:

[0213] If the target area is the first area, the first area is used to store order data; a first target sub-area is determined in the first area; different sub-areas in the queue area are used to store order data of different users;

[0214] The processing corresponding to the first request is performed on the target data in the first target sub-area.

[0215] In some embodiments, the processing unit 1203 is further configured to:

[0216] If the target area is the second area, the second area is used to store static data; determining the access status of the target data;

[0217] Based on the access status, determining a second target sub-area in the second area; different sub-areas in the second area are used to store static data with different access statuses;

[0218] The processing corresponding to the first request is performed on the target data in the second target sub-area.

[0219] In some embodiments, the processing unit 1203 is further configured to:

[0220] If the access state is a first state representing hotspot data, determining the first sub-region as the second target sub-region; the access frequency of the hotspot data is greater than or equal to a first frequency threshold; the first sub-region caches data through the first middleware;

[0221] If the access state is the second state characterized by non-hotspot data, the second sub-area is determined to be the second target sub-area; the access frequency of the non-hotspot data is less than the first frequency threshold; the second sub-area caches data through the second middleware; the data processing rate of the second middleware is less than the data processing rate of the first middleware.

[0222] In some embodiments, the processing unit 1203 is further configured to:

[0223] If the target area is the third area, the third area is used to store state data; and the processing corresponding to the first request is performed on the target data in the third area through a Bloom filter.

[0224] In some embodiments, the processing unit 1203 is further configured to:

[0225] If the target area is the fourth area, the fourth area is used to store counting data; the first request is a write request, and it is determined that the target data is greater than or equal to 1, and the target data is written into the fourth area.

[0226] It should be noted that the data processing device provided in the embodiment of the present application includes the various units included, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0227] The description of the above device embodiment is similar to the description of the above method embodiment and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of this application, please refer to the description of the method embodiment of this application for understanding.

[0228] It should be noted that, in the embodiment of the present application, if the above-mentioned data processing method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific combination of hardware and software.

[0229] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the steps in the data processing method provided in the above embodiment are implemented.

[0230] The following combination Figure 13 The structure of the electric device 130 is described below.

[0231] In one example, if Figure 13As shown, the electronic device 130 includes: a processor 1301, at least one communication bus 1302, a user interface 1303, at least one external communication interface 1304, and a memory 1305. The communication bus 1302 is configured to enable communication between these components. The user interface 1303 may include a display screen, and the external communication interface 1304 may include a standard wired interface and a wireless interface.

[0232] The memory 1305 is configured to store instructions and applications executable by the processor 1301, and can also cache data to be processed or processed by the processor 1301 and various modules in the electronic device (for example, image data, audio data, voice communication data and video communication data), which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0233] In a fourth aspect, an embodiment of the present application provides a storage medium, that is, a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the data processing method provided in the above embodiment are implemented.

[0234] It should be noted that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0235] It should be understood that “one embodiment” or “an embodiment” mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, “in one embodiment” or “in some embodiments” appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. The above-mentioned serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments.

[0236] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0237] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0238] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.

[0239] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0240] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, and other media that can store program codes.

[0241] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0242] The above is merely an embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, which is applied to the scenario of medical applications in Internet hospitals, is characterized in that: The method comprises: Receiving a first request through a first task, and determining a data type of target data to be processed by the first request; the data type includes at least one of the following: order data, static data, status data, and count data; Determine a target area and a target cache middleware from at least one area included in the cache based on the data type of the target data; different areas in the at least one area are used to store data of different data types; the at least one area includes at least one of the following: a first area for storing order data, a second area for storing static data, a third area for storing status data, and a fourth area for storing count data; the target cache middleware corresponding to the second area includes a first middleware for processing hotspot data and a second middleware for processing non-hotspot data; A second task is used to execute processing corresponding to the first request on the target data in the target area based on the target cache middleware; the first task is different from the second task.

2. The method according to claim 1, characterized in that If the target area is the first area, the first area is used to store order data; The performing, in the target area, processing corresponding to the first request on the target data based on the target cache middleware includes: Determining a first target sub-area in the first area; different sub-areas in the first area are used to store order data of different users; In the first target sub-area, based on the target cache middleware, a process corresponding to the first request is performed on the target data.

3. The method according to claim 1, characterized in that If the target area is the second area, the second area is used to store static data; The performing, in the target area, processing corresponding to the first request on the target data based on the target cache middleware includes: determining an access status of the target data; Based on the access status, determining a second target sub-area in the second area; different sub-areas in the second area are used to store static data with different access statuses; In the second target sub-area, based on the target cache middleware, a process corresponding to the first request is executed on the target data.

4. The method according to claim 3, characterized in that The determining, based on the access status, a second target sub-area in the second area includes: If the access state is a first state representing hotspot data, determining the first sub-region as the second target sub-region; the access frequency of the hotspot data is greater than or equal to a first frequency threshold; the first sub-region caches data through the first middleware; If the access state is the second state characterized by non-hotspot data, the second sub-area is determined to be the second target sub-area; the access frequency of the non-hotspot data is less than the first frequency threshold; the second sub-area caches data through the second middleware; the data processing rate of the second middleware is less than the data processing rate of the first middleware.

5. The method according to claim 1, wherein If the target area is a third area, the third area is used to store state data; performing processing corresponding to the first request on the target data based on the target cache middleware in the target area includes: The processing corresponding to the first request is performed on the target data in the third area based on the target cache middleware through the Bloom filter.

6. The method according to claim 1, characterized in that If the target area is the fourth area, the fourth area is used to store counting data; The first request is a write request, and performing processing corresponding to the first request on the target data in the target area includes: It is determined that the target data is greater than or equal to 1, and the target data is written into the fourth area based on the target cache middleware.

7. A data processing device, which is applied to a medical application program of an Internet hospital, characterized in that: The device comprises: a receiving unit, configured to receive a first request through a first task, and determine a data type of target data to be processed by the first request; the data type comprising at least one of the following: order data, static data, status data, and count data; A determination unit is configured to determine a target area and a target cache middleware from at least one area included in the cache based on a data type of the target data; different areas in the at least one area are configured to store data of different data types; the at least one area includes at least one of the following: a first area for storing order data, a second area for storing static data, a third area for storing status data, and a fourth area for storing count data; the target cache middleware corresponding to the second area includes a first middleware for processing hotspot data and a second middleware for processing non-hotspot data; A processing unit is configured to execute processing corresponding to the first request on the target data based on the target cache in the target area through a second task; the first task is different from the second task.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the data processing method according to any one of claims 1 to 6 is implemented.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

Images