Method, apparatus, electronic device, and storage medium for data processing

Abstract

The application discloses a data processing method and device, electronic equipment and storage medium, and relates to the technical field of computers. A specific implementation manner of the method comprises the following steps: receiving a data request, obtaining a data identifier in the data request, querying corresponding target data from a cache, returning the target data, and triggering an update operation of the target data; in response to the update operation, obtaining an expiration timestamp of the target data and an update threshold, wherein the update threshold is determined from an update time interval corresponding to the expiration timestamp based on an attribute parameter of the target data; calculating a time length between the expiration timestamp and a current time, determining that the time length is less than the update threshold, and calling a preset data update thread to update the target data in the cache. The implementation manner can solve the problem that, in a high-concurrency scenario, data in the cache reaches an expiration time and then data is updated, which reduces the efficiency of data request and response and causes the performance of a business system to decrease.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the rapid development of the internet, high concurrency has become a frequent scenario for business systems processing business data, and expiration-refresh caching mechanisms are a common approach in high-concurrency scenarios. Expiration-refresh caching mechanisms involve setting an expiration time for data stored in the cache, and updating the data when the expiration time is reached. However, during high-concurrency stress testing of business systems, it was found that even with sufficient optimization of downstream data sources to ensure timely updates, the efficiency of data requests and responses still decreases after the cached data expires, leading to a decline in the performance of the business system. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data processing method, apparatus, electronic device, and storage medium that can solve the problem that updating data in the cache when it expires in high-concurrency scenarios reduces the efficiency of data requests and responses, leading to a decline in the performance of business systems.

[0004] To achieve the above objectives, according to one aspect of the present invention, a data processing method is provided.

[0005] An embodiment of the present invention provides a data processing method comprising: receiving a data request; obtaining a data identifier in the data request to query corresponding target data from a cache; returning the target data; and triggering an update operation on the target data; responding to the update operation; obtaining an expiration timestamp and an update threshold of the target data, wherein the update threshold is determined based on the attribute parameters of the target data from an update time interval corresponding to the expiration timestamp; calculating the duration between the expiration timestamp and the current time; determining that the duration is less than the update threshold; and invoking a preset data update thread to update the target data in the cache.

[0006] In one embodiment, before triggering the update operation of the target data, the method further includes:

[0007] Generate an expiration timestamp for the target data, query the time period to which the expiration timestamp belongs, and determine the update time interval corresponding to the time period to which the expiration timestamp belongs as the update time interval corresponding to the expiration timestamp, wherein the time period includes idle time periods and non-idle time periods;

[0008] Obtain the attribute parameters of the target data, and determine the update threshold from the update time interval corresponding to the expiration timestamp based on the attribute parameters.

[0009] In yet another embodiment, determining the update threshold from the update time interval corresponding to the expired timestamp based on the attribute parameters includes:

[0010] Based on the business level of the target data, the acquisition duration of the target data, and / or the priority of the target data, the corresponding update time is queried from the update time interval to determine the update threshold.

[0011] In yet another embodiment, determining the update threshold from the update time interval corresponding to the expired timestamp based on the attribute parameters includes:

[0012] A preset conversion model is invoked to calculate the conversion code corresponding to the data identifier, and the update threshold is determined from the update time interval based on the conversion code.

[0013] In yet another embodiment, the step of invoking a preset data update thread to update the target data in the cache includes:

[0014] Get the update task lock;

[0015] In response to a successful acquisition of the update task lock, a preset data update thread is invoked to update the target data in the cache; or...

[0016] In response to the failure to acquire the update task lock, a pre-defined update retry operation is performed.

[0017] In yet another embodiment, the update retry operation includes:

[0018] Generate an update request for the target data and store the update request in the update retry queue;

[0019] Based on a preset period, the update request with the smallest expiration timestamp is selected from the update retry queue and processed accordingly.

[0020] In yet another embodiment, it further includes:

[0021] Receive a data update notification message from the data source and send a data retrieval request to the data source;

[0022] Receive update data sent by the data source and update the cache with the update data;

[0023] Generate an expiration timestamp for the updated data to determine the update time interval of the updated data;

[0024] Obtain the attribute parameters of the updated data, determine the update threshold from the update time interval of the updated data based on the attribute parameters of the updated data, and store the expiration timestamp and update threshold of the updated data.

[0025] To achieve the above objectives, according to another aspect of the present invention, a data processing apparatus is provided.

[0026] An embodiment of the present invention provides a data processing apparatus comprising: a triggering unit, configured to receive a data request, acquire a data identifier from the data request, query corresponding target data from a cache, return the target data, and trigger an update operation on the target data; an acquisition unit, configured to, in response to the update operation, acquire an expiration timestamp and an update threshold of the target data, wherein the update threshold is determined based on attribute parameters of the target data from an update time interval corresponding to the expiration timestamp; and an update unit, configured to calculate the duration between the expiration timestamp and the current time, determine that the duration is less than the update threshold, and invoke a preset data update thread to update the target data in the cache.

[0027] In one embodiment, the apparatus further includes:

[0028] The generation unit is used to generate an expiration timestamp of the target data, query the time period to which the expiration timestamp belongs, and determine the update time interval corresponding to the time period to which the expiration timestamp belongs as the update time interval corresponding to the expiration timestamp, wherein the time period includes idle time periods and non-idle time periods;

[0029] A determining unit is used to obtain attribute parameters of the target data, so as to determine the update threshold from the update time interval corresponding to the expiration timestamp based on the attribute parameters.

[0030] In yet another embodiment, the determining unit is specifically used for:

[0031] Based on the business level of the target data, the acquisition duration of the target data, and / or the priority of the target data, the corresponding update time is queried from the update time interval to determine the update threshold.

[0032] In yet another embodiment, the determining unit is specifically used for:

[0033] A preset conversion model is invoked to calculate the conversion code corresponding to the data identifier, and the update threshold is determined from the update time interval based on the conversion code.

[0034] In yet another embodiment, the updating unit is specifically used for:

[0035] Get the update task lock;

[0036] In response to a successful acquisition of the update task lock, a preset data update thread is invoked to update the target data in the cache; or...

[0037] In response to the failure to acquire the update task lock, a pre-defined update retry operation is performed.

[0038] In yet another embodiment, the updating unit is specifically used for:

[0039] Generate an update request for the target data and store the update request in the update retry queue;

[0040] Based on a preset period, the update request with the smallest expiration timestamp is selected from the update retry queue and processed accordingly.

[0041] In yet another embodiment, the receiving unit is further configured to receive a data update notification message sent by the data source and send a data acquisition request to the data source;

[0042] The update unit is further configured to receive update data sent by the data source and update the cache with the update data;

[0043] The generation unit is also used to generate an expiration timestamp of the updated data to determine the update time range of the updated data;

[0044] The determining unit is further configured to acquire attribute parameters of the updated data, determine an update threshold from the update time interval of the updated data based on the attribute parameters of the updated data, and store the expiration timestamp and update threshold of the updated data.

[0045] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.

[0046] An electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided in the embodiment of the present invention.

[0047] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided.

[0048] An embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method provided in the embodiment of the present invention.

[0049] One embodiment of the above invention has the following advantages or beneficial effects: In this embodiment of the invention, after receiving a data request, the target data can be queried from the cache and then returned, while triggering an update operation for the target data; in response to the update operation, the expiration timestamp and update threshold of the target data can be obtained, and after determining that the time between the expiration timestamp and the current time is less than the update threshold, a preset data update thread is called to update the target data in the cache. In this embodiment of the invention, an update threshold is set for data in the cache. When querying target data from the cache, the expiration timestamp of the target data and the update threshold can be used to determine whether the target data is nearing its expiration time. That is, whether the time between the expiration timestamp and the current time is less than the update threshold. If the time is less than the update threshold, it means that the target data is nearing its expiration time, and the target data can be updated in a timely manner. This avoids waiting for the target data to reach its expiration time before updating, keeping the data in the cache in an unexpired state and preventing data expiration from affecting the efficiency of data requests and responses, thus improving the performance of the business system. At the same time, in this embodiment of the invention, a corresponding update time interval is set for the expiration timestamp, and then the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data. This allows each piece of data in the cache to have a different update threshold, avoiding the situation where the expiration time and update threshold of each piece of data in the cache are the same, which would lead to too much data being updated at the same time and degrade the performance of the database and the business system.

[0050] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0051] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0052] Figure 1 This is a schematic diagram of a system architecture for a data processing system according to an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of a main flow of a data processing method according to an embodiment of the present invention;

[0054] Figure 3 This is a schematic diagram of another main flow of a data processing method according to an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram of the main units of a data processing apparatus according to an embodiment of the present invention;

[0056] Figure 5 This is yet another exemplary system architecture diagram to which embodiments of the present invention can be applied;

[0057] Figure 6This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present invention. Detailed Implementation

[0058] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0059] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0060] This invention provides a data processing system that can be used for processing cached data, specifically for updating cached data in high-concurrency scenarios (typically referring to systems designed to handle many requests simultaneously). High-concurrency scenarios can include various types, such as high-concurrency data requests in business processing or high-concurrency stress testing. Stress testing is a method for establishing system stability, typically conducted outside the system's normal operating range to examine its functional limits and potential problems. It primarily tests the server's capacity, including user capacity (how many users can operate simultaneously without significantly affecting quality) and traffic capacity.

[0061] Specifically, such as Figure 1 As shown, the data processing system can be a business system, including an API (Application Programming Interface) controller, a cache controller, and a cache, such as Redis (Remote Dictionary Server, an open-source, ANSI C-written, network-enabled, in-memory or persistent log-structured key-value database that provides APIs in multiple languages ​​and is commonly used for high-performance caching). The API controller can communicate with the data requester to receive data requests and forward them to the cache controller for processing, and then send the data returned by the cache controller back to the data requester. The cache controller can process data requests issued by the API controller and manage the data in the cache. The cache can store various types of data. In this embodiment of the invention, the cache stores data using a key-value pair, where the key represents a data identifier and the value represents the corresponding data.

[0062] Since the expiration time of data in the cache is usually in milliseconds, it is not convenient for comparing whether data is about to expire. Therefore, in this embodiment of the invention, an expiration timestamp can be added to each piece of data. For example, the Redis cache model can be: redis.set(key, (stamp, value), timeout), where stamp represents the expiration timestamp of value, and timeout represents the expiration time of value. The stamp can be calculated based on the expiration time when data is stored. The expiration time can be preset or calculated based on the set storage duration.

[0063] To avoid updating cached data only after it has expired, this embodiment of the invention sets an update threshold for each piece of data. The update threshold indicates the remaining time before the data expires. When the remaining time before the data expires is less than the update threshold, it means that the data is about to expire and can be updated first. This achieves timely updates to the cached data, improves the validity of the cached data, and thus improves the consistency and stability of the cached data. It also prevents the cached data from expiring and reducing the efficiency of the business system in processing data requests and responses.

[0064] Since updated data needs to be retrieved from the data source before the data expires, if a large amount of data is updated at the same time, a large amount of data will perform read and write operations on the cache at the same time, which may cause problems such as cache breakdown. Therefore, in this embodiment of the invention, different update thresholds can be set for each data to avoid centralized data updates.

[0065] Specifically, the update threshold for each piece of data in the cache can be determined based on the expiration timestamp and the attribute parameters of the data.

[0066] An expiration timestamp indicates when data expires. To reduce data update frequency, updates are typically performed when data is close to expiration, so the update threshold is usually not very long. Data updates usually require calling an API to retrieve new data, and data updates typically take longer when the business system is busy than when it is idle. Therefore, in this embodiment of the invention, idle and non-idle time periods can be divided based on the idle depth of the business system, and corresponding update time intervals can be set for each idle and non-idle time period. The update time interval for the idle time period is longer than that for the non-idle time period. Thus, when determining the update threshold, the time period to which the expiration timestamp belongs can be determined first, and then the update threshold can be determined from the update time interval corresponding to that time period. This ensures that data whose expiration time belongs to the idle time period has a smaller update threshold, while data whose expiration time belongs to the non-idle time period has a larger update threshold, thus ensuring timely data updates while reducing the frequency of data updates. For example, the update time interval for the idle time period can be 5-10 seconds, and the update time interval for the non-idle time period can be 11-20 seconds.

[0067] Data attribute parameters can include various types, such as data business level, data acquisition time, data priority, etc. Business level can indicate the importance of the data to the corresponding business; data acquisition time can indicate the time spent acquiring the data; and data priority can indicate the importance of the data. In this embodiment of the invention, a mapping relationship between one or more of the above attribute parameters and each value in the update time interval can be set, and then the mapping relationship can be queried based on each attribute parameter to obtain the corresponding update time from the update time interval. Additionally, data attribute parameters can also include data identifiers. In this embodiment of the invention, a conversion model can be set to convert each data identifier into a preset format conversion code, and a mapping relationship between the conversion code and each value in the update time interval can be pre-set, so that the corresponding update time can be obtained from the update time interval based on the conversion code. For example, the conversion model can be a hash algorithm.

[0068] This invention provides a data processing method, which can be performed by... Figure 1 The data processing system shown executes, such as Figure 2 As shown, the method includes:

[0069] S201: Receive a data request, obtain the data identifier in the data request, query the corresponding target data from the cache, return the target data, and trigger the target data update operation.

[0070] The data request includes a data identifier. The API controller receives the data request and forwards it to the cache controller. The cache controller retrieves the data identifier from the data request and then queries the cache for the corresponding target data. After retrieving the target data, the cache controller forwards it to the API controller, which then returns the target data to the requesting client. Simultaneously, the cache controller can trigger an update operation for the target data to ensure timely updates.

[0071] It should be noted that, due to reasons such as not being stored or having expired, the cache may not be able to find the target data corresponding to the data identifier. In this case, the data retrieval thread can be called to retrieve the target data from the database or third-party data source, and then the retrieved target data can be returned. At the same time, the data is updated in the cache and stored, and the corresponding expiration timestamp and update threshold are set for the data.

[0072] S202: In response to the update operation, obtain the expiration timestamp and update threshold of the target data, wherein the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data.

[0073] Once the update operation is triggered, an update process can be executed for the target data. The expiration timestamp and update threshold of the target data are predetermined. This step first obtains the expiration timestamp and update threshold of the target data to determine if the target data is nearing expiration and requires an update.

[0074] S203: Calculate the duration between the expiration timestamp and the current time, determine if the duration is less than the update threshold, and call the preset data update thread to update the target data in the cache.

[0075] The expiration timestamp represents the expiration time of the target data. Calculating the duration between the expiration timestamp and the current time allows us to determine the remaining validity period of the target data before its expiration. In this embodiment, an update threshold is used to determine whether data needs to be updated. If the calculated duration is less than the update threshold, it indicates that the target data is nearing expiration, and a preset data update thread can be invoked to update the cached target data. If the calculated duration is not less than the update threshold, it indicates that the target data is not nearing expiration, and no update is required.

[0076] In this embodiment of the invention, the data update thread can be an asynchronous thread. To avoid excessive data updates simultaneously, which could lead to high-concurrency cache rebuilding risks, a mutex lock, or update task lock, can be set for data updates. Each time data is updated, the update task lock must be acquired first. If the acquisition is successful, a preset data update thread is invoked to perform the data update; otherwise, the process waits to execute a preset update retry operation. This avoids excessive data updates simultaneously, which could lead to a large number of update requests directly reaching the database or data source, causing database access bottlenecks.

[0077] Specifically, the update retry operation can be executed as follows: generate an update request for the target data and store the update request in the update retry queue; based on a preset period, select the update request with the smallest expiration timestamp from the update retry queue and process the update request with the smallest expiration timestamp.

[0078] In this embodiment of the invention, a pre-defined update retry queue is provided. If acquiring the update task lock fails during data update, an update request can be generated and stored in the update retry queue. Then, each update request is processed sequentially based on its corresponding data expiration time, ensuring that data nearing expiration is updated promptly and preventing data in the cache from expiring. The cycle duration can be set based on specific scenarios.

[0079] In this embodiment of the invention, in order to update the changed data in a timely manner, the data source can be set to actively send a data update prompt message when the data is updated. Therefore, after receiving the data update prompt message sent by the data source, a data acquisition request can be sent directly to the data source to obtain the updated data in a timely manner, and then the updated data is updated in the cache. At the same time, an expiration timestamp of the updated data is generated. Then, the update threshold of the updated data can be determined based on the expiration timestamp of the updated data, and the expiration timestamp and update threshold of the updated data can be stored for the next data update.

[0080] In this embodiment of the invention, an update threshold is set for data in the cache. When querying data from the cache, the expiration timestamp of the target data and the update threshold can be used to determine whether the target data is nearing its expiration time. That is, whether the time between the expiration timestamp and the current time is less than the update threshold. If the time is less than the update threshold, it means that the target data is nearing its expiration time, and the target data can be updated in a timely manner. This avoids waiting for the target data to reach its expiration time before updating, keeping the data in the cache in an unexpired state and preventing data expiration from affecting the efficiency of data requests and responses, thus improving the performance of the business system. At the same time, in this embodiment of the invention, a corresponding update time interval is set for the expiration timestamp, and then the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data. This allows each piece of data in the cache to have a different update threshold, avoiding the situation where the expiration time and update threshold of each piece of data in the cache are the same, which would lead to too much data being updated at the same time and degrade the performance of the database and the business system.

[0081] The following is combined Figure 2 The embodiments shown in this paper provide another data processing method, such as... Figure 3 As shown, the method includes:

[0082] S301: Receive data request and obtain the data identifier in the data request.

[0083] S302: Determine whether the target data corresponding to the data identifier is stored in the cache. If yes, proceed to step S303; otherwise, call the data acquisition thread to obtain the target data, return the target data, update the target data to the cache, and set the corresponding expiration timestamp and update threshold.

[0084] S303: Returns the target data, triggering an update operation for the target data.

[0085] S304: In response to an update operation, obtain the expiration timestamp and update threshold of the target data.

[0086] S305: Calculate the duration between the expired timestamp and the current time.

[0087] S306: Determine if the duration is less than the update threshold. If yes, proceed to step S307; otherwise, do not perform the update.

[0088] S307: Acquire the update task lock. If the update task lock is successfully acquired, call the preset data update thread to update the target data in the cache; or, if the update task lock is not acquired, execute the preset update retry operation.

[0089] It should be noted that the data processing principle in the embodiments of the present invention is the same as... Figure 2The data processing principles in the illustrated embodiments are the same, and will not be repeated here.

[0090] In this embodiment of the invention, an update threshold is set for data in the cache. When querying data from the cache, the expiration timestamp of the target data and the update threshold can be used to determine whether the target data is nearing its expiration time. That is, whether the time between the expiration timestamp and the current time is less than the update threshold. If the time is less than the update threshold, it means that the target data is nearing its expiration time, and the target data can be updated in a timely manner. This avoids waiting until the target data reaches its expiration time before updating, keeping the data in the cache in an unexpired state and preventing data expiration from affecting the efficiency of data requests and responses, thus improving the performance of the business system. At the same time, in this embodiment of the invention, a corresponding update time interval is set for the expiration timestamp, and then the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data. This allows each piece of data in the cache to have a different update threshold, thereby avoiding the situation where the expiration time and update threshold of each piece of data in the cache are the same, which would lead to too much data being updated at the same time and degrading the performance of the database and the business system.

[0091] To address the problems existing in the prior art, embodiments of the present invention provide a data processing apparatus 400, such as... Figure 4 As shown, the device 400 includes:

[0092] The triggering unit is used to receive a data request, obtain the data identifier in the data request, query the corresponding target data from the cache, return the target data, and trigger the update operation of the target data;

[0093] An acquisition unit is configured to, in response to the update operation, acquire the expiration timestamp and update threshold of the target data, wherein the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data;

[0094] The update unit is used to calculate the duration between the expiration timestamp and the current time, determine that the duration is less than the update threshold, and call a preset data update thread to update the target data in the cache.

[0095] It should be understood that the manner in which the embodiments of the present invention are implemented is different from the implementation method. Figure 2 The methods used in the illustrated embodiments are the same and will not be repeated here.

[0096] In one implementation of this invention, the device 400 further includes:

[0097] The generation unit is used to generate an expiration timestamp of the target data, query the time period to which the expiration timestamp belongs, and determine the update time interval corresponding to the time period to which the expiration timestamp belongs as the update time interval corresponding to the expiration timestamp, wherein the time period includes idle time periods and non-idle time periods;

[0098] A determining unit is used to obtain attribute parameters of the target data, so as to determine the update threshold from the update time interval corresponding to the expiration timestamp based on the attribute parameters.

[0099] In another implementation of this invention, the determining unit 400 is specifically used for:

[0100] Based on the business level of the target data, the acquisition duration of the target data, and / or the priority of the target data, the corresponding update time is queried from the update time interval to determine the update threshold.

[0101] In another implementation of this invention, the determining unit 400 is specifically used for:

[0102] A preset conversion model is invoked to calculate the conversion code corresponding to the data identifier, and the update threshold is determined from the update time interval based on the conversion code.

[0103] In another implementation of this invention, the update unit 403 is specifically used for:

[0104] Get the update task lock;

[0105] In response to a successful acquisition of the update task lock, a preset data update thread is invoked to update the target data in the cache; or...

[0106] In response to the failure to acquire the update task lock, a pre-defined update retry operation is performed.

[0107] In another implementation of this invention, the update unit 403 is specifically used for:

[0108] Generate an update request for the target data and store the update request in the update retry queue;

[0109] Based on a preset period, the update request with the smallest expiration timestamp is selected from the update retry queue and processed accordingly.

[0110] In another implementation of this invention, the receiving unit 401 is further configured to receive a data update notification message sent by the data source and send a data acquisition request to the data source.

[0111] The update unit 403 is further configured to receive update data sent by the data source and update the cache with the update data;

[0112] The generation unit is also used to generate an expiration timestamp of the updated data to determine the update time range of the updated data;

[0113] The determining unit is further configured to acquire attribute parameters of the updated data, determine an update threshold from the update time interval of the updated data based on the attribute parameters of the updated data, and store the expiration timestamp and update threshold of the updated data.

[0114] It should be understood that the manner in which the embodiments of the present invention are implemented is different from the implementation method. Figure 2 or Figure 3 The methods used in the illustrated embodiments are the same and will not be repeated here.

[0115] In this embodiment of the invention, an update threshold is set for data in the cache. When querying data from the cache, the expiration timestamp of the target data and the update threshold can be used to determine whether the target data is nearing its expiration time. That is, whether the time between the expiration timestamp and the current time is less than the update threshold. If the time is less than the update threshold, it means that the target data is nearing its expiration time, and the target data can be updated in a timely manner. This avoids waiting for the target data to reach its expiration time before updating, keeping the data in the cache in an unexpired state and preventing data expiration from affecting the efficiency of data requests and responses, thus improving the performance of the business system. At the same time, in this embodiment of the invention, a corresponding update time interval is set for the expiration timestamp, and then the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data. This allows each piece of data in the cache to have a different update threshold, avoiding the situation where the expiration time and update threshold of each piece of data in the cache are the same, which would lead to too much data being updated at the same time and degrade the performance of the database and the business system.

[0116] According to embodiments of the present invention, an electronic device and a readable storage medium are also provided.

[0117] An electronic device according to an embodiment of the present invention includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the data processing method provided in the embodiment of the present invention.

[0118] Figure 5 An exemplary system architecture 500 is shown, in which a data processing method or data processing apparatus to which embodiments of the present invention can be applied is illustrated.

[0119] like Figure 5As shown, system architecture 500 may include terminal devices 501, 502, and 503, a network 504, and a server 505. Network 504 serves as the medium for providing communication links between terminal devices 501, 502, and 503 and server 505. Network 504 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0120] Users can use terminal devices 501, 502, and 503 to interact with server 505 via network 504 to receive or send messages, etc. Various client applications can be installed on terminal devices 501, 502, and 503.

[0121] Terminal devices 501, 502, and 503 can be, but are not limited to, smartphones, tablets, laptops, and desktop computers, etc.

[0122] Server 505 can be a server that provides various services. The server can analyze and process the received data requests and other data, and feed back the processing results (such as target data - just an example) to the terminal device.

[0123] It should be noted that the data processing method provided in the embodiments of the present invention is generally executed by server 505, and correspondingly, the data processing device is generally located in server 505.

[0124] It should be understood that Figure 5 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0125] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing embodiments of the present invention. Figure 6 The computer system shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0126] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0127] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0128] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0129] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0130] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a unit, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0131] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor can be described as including a receiving unit, an acquiring unit, and an updating unit. The names of these units do not necessarily limit the specific unit; for example, a receiving unit can also be described as "a unit that performs the functions of a receiving unit."

[0132] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform the data processing method provided by the present invention.

[0133] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A data processing method, characterized in that, include: Receive a data request, obtain the data identifier in the data request, query the corresponding target data from the cache, return the target data, and trigger an update operation for the target data; In response to the update operation, the expiration timestamp and update threshold of the target data are obtained, wherein the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data; Calculate the duration between the expiration timestamp and the current time, determine if the duration is less than the update threshold, and then call a preset data update thread to update the target data in the cache. Before triggering the update operation of the target data, the following is also included: Generate an expiration timestamp for the target data, query the time period to which the expiration timestamp belongs, and determine the update time interval corresponding to the time period to which the expiration timestamp belongs as the update time interval corresponding to the expiration timestamp. The time period includes idle time periods and non-idle time periods. Set corresponding update time intervals for idle time periods and non-idle time periods, so that the update time interval corresponding to the idle time period is greater than the update time interval corresponding to the non-idle time period. Obtain the attribute parameters of the target data, and determine the update threshold from the update time interval corresponding to the expiration timestamp based on the attribute parameters.

2. The method according to claim 1, characterized in that, Determining the update threshold from the update time interval corresponding to the expired timestamp based on the attribute parameters includes: Based on the business level of the target data, the acquisition duration of the target data, and / or the priority of the target data, the corresponding update time is queried from the update time interval to determine the update threshold.

3. The method according to claim 1, characterized in that, Determining the update threshold from the update time interval corresponding to the expired timestamp based on the attribute parameters includes: A preset conversion model is invoked to calculate the conversion code corresponding to the data identifier, and the update threshold is determined from the update time interval based on the conversion code.

4. The method according to claim 1, characterized in that, The step of invoking a preset data update thread to update the target data in the cache includes: Get the update task lock; In response to a successful acquisition of the update task lock, a preset data update thread is invoked to update the target data in the cache; or... In response to the failure to acquire the update task lock, a pre-defined update retry operation is performed.

5. The method according to claim 4, characterized in that, The update retry operation includes: Generate an update request for the target data and store the update request in the update retry queue; Based on a preset period, the update request with the smallest expiration timestamp is selected from the update retry queue and processed accordingly.

6. The method according to claim 1, characterized in that, Also includes: Receive a data update notification message from the data source and send a data retrieval request to the data source; Receive update data sent by the data source and update the cache with the update data; Generate an expiration timestamp for the updated data to determine the update time interval of the updated data; Obtain the attribute parameters of the updated data, determine the update threshold from the update time interval of the updated data based on the attribute parameters of the updated data, and store the expiration timestamp and update threshold of the updated data.

7. A data processing apparatus, characterized in that, include: The triggering unit is used to receive a data request, obtain the data identifier in the data request, query the corresponding target data from the cache, return the target data, and trigger the update operation of the target data; An acquisition unit is configured to, in response to the update operation, acquire the expiration timestamp and update threshold of the target data, wherein the update threshold is determined from the update time interval corresponding to the expiration timestamp based on the attribute parameters of the target data; The update unit is used to calculate the duration between the expiration timestamp and the current time, determine that the duration is less than the update threshold, and call a preset data update thread to update the target data in the cache. The device further includes: The generation unit is used to generate an expiration timestamp of the target data, query the time period to which the expiration timestamp belongs, and determine the update time interval corresponding to the time period to which the expiration timestamp belongs as the update time interval corresponding to the expiration timestamp. The time period includes idle time periods and non-idle time periods. The corresponding update time intervals are set for idle time periods and non-idle time periods, so that the update time interval corresponding to the idle time period is greater than the update time interval corresponding to the non-idle time period. A determining unit is used to obtain attribute parameters of the target data, so as to determine the update threshold from the update time interval corresponding to the expiration timestamp based on the attribute parameters.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images