Data acquisition method, device and program product

Abstract

The embodiment of the invention relates to a data acquisition method and device and a program product, and the method comprises the steps: enabling a plurality of threads to respectively request to acquire a distributed lock under the condition that data acquisition requests initiated by the plurality of threads for target data in a target system are received; accessing the target system through a first thread in the plurality of threads to obtain target data in the target system, and caching the target data to a specified database; the first thread is a thread requesting to successfully acquire the distributed lock in the plurality of threads; obtaining the target data from the specified database through a second thread in the plurality of threads; the second thread is the thread which fails to request to acquire the distributed lock in the threads. According to the embodiment of the invention, the situation that a plurality of threads access the target system in order to obtain data can be effectively improved, so that the problems of frequent calling of external system interfaces, resource competition and the like caused by the situation can be well relieved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technologies, and in particular, to a data acquisition method, device, and program product. Background Art

[0002] In modern information systems, high-concurrency access to external systems has become a prevalent challenge. In high-concurrency scenarios, it often occurs that multiple threads simultaneously access the same external system to obtain required data, which may cause many problems such as frequent external system interface calls and resource contention. Summary of the Invention

[0003] To solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a data acquisition method, device, and program product.

[0004] An embodiment of the present disclosure provides a data acquisition method, which includes: when receiving data acquisition requests initiated by multiple threads for target data in a target system, causing the multiple threads to respectively request to obtain a distributed lock; accessing the target system through a first thread among the multiple threads to obtain the target data in the target system, and caching the target data into a specified database; where the first thread is the thread that successfully requests to obtain the distributed lock among the multiple threads; obtaining the target data from the specified database through a second thread among the multiple threads; where the second thread is the thread that fails to request to obtain the distributed lock among the multiple threads.

[0005] Optionally, the obtaining the target data from the specified database through a second thread among the multiple threads includes: monitoring the current state of the distributed lock; where the current state includes a held state or a released state; in response to the current state being the released state, obtaining the target data from the specified database through a second thread among the multiple threads.

[0006] Optionally, the method further includes: in response to meeting a preset lock release condition, releasing the distributed lock through the first thread.

[0007] Optionally, the lock release condition includes one or more of the following: all the target data has been cached into the specified database, the duration for which the first thread holds the distributed lock reaches a preset lock holding duration threshold, and it is detected that the first thread is abnormal.

[0008] Optionally, the method further includes: determining the lock holding duration threshold based on the number of the multiple threads.

[0009] Optionally, the method further includes: in response to detecting invalid data in the specified database, obtaining valid data corresponding to the invalid data from the target system, and replacing the invalid data in the specified database with the valid data; and / or, in response to receiving a pre-fetch instruction for backup data, obtaining the backup data from the target system and pre-storing the backup data in the specified database.

[0010] Optionally, accessing the target system through the first thread among the multiple threads to obtain target data in the target system includes: querying from the specified database through the first thread among the multiple threads whether the target data in the target system is cached; in response to querying the first data in the target data from the specified database, obtaining the first data from the specified database through the first thread, and accessing the target system to obtain the second data in the target data from the target system; where the first data is at least part of the target data, and the second data is the data in the target data other than the first data; in response to not querying the target data from the specified database, accessing the target system through the first thread to obtain the target data in the target system.

[0011] Optionally, the method further includes: in response to the second thread not successfully obtaining the target data from the specified database, instructing the second thread to request to obtain the distributed lock again.

[0012] An embodiment of the present disclosure further provides an electronic device, including: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the data acquisition method provided by the embodiment of the present disclosure.

[0013] An embodiment of the present disclosure further provides a computer program product, including a computer program, where the computer program implements the data acquisition method provided by the embodiment of the present disclosure when executed by a processor.

[0014] In the above technical solution provided by the embodiments of the present disclosure, when receiving data acquisition requests initiated by multiple threads for target data in a target system, it is possible to enable the multiple threads to respectively request to acquire a distributed lock. Only the threads that successfully request to acquire the distributed lock among the multiple threads can access the target system to acquire the target data in the target system, and cache the target data in a specified database. The threads that fail to request to acquire the distributed lock among the multiple threads can directly acquire the target data from the specified database. The above method can not only effectively ensure that all multiple threads can acquire data, but also effectively improve the situation where multiple threads access the target system to acquire data, thereby better alleviating problems such as frequent external system interface calls and resource contention caused thereby.

[0015] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 It is a schematic flowchart of a data acquisition method provided by an embodiment of the present disclosure;

[0019] Figure 2 It is a schematic flowchart of a data acquisition process provided by an embodiment of the present disclosure;

[0020] Figure 3 It is a schematic structural diagram of a data acquisition device provided by an embodiment of the present disclosure;

[0021] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] In order to be able to more clearly understand the above objects, features and advantages of the present disclosure, the solution of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0023] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure. However, the present disclosure may also be implemented in other ways different from those described herein. Obviously, the embodiments in the specification are only a part of the embodiments of the present disclosure, rather than all the embodiments.

[0024] Currently, there are many applications and services that need to frequently call external systems such as third-party databases and microservices to obtain the latest data. However, such frequent calls will not only increase the load on the external systems, but may also cause problems such as network latency and service instability. Therefore, effectively managing concurrent requests and reducing repeated calls to external systems become the key to improving system performance. The inventors have found through research that in high-concurrency scenarios, it often occurs that multiple threads simultaneously access the same external system to obtain the required data, which will cause many problems, which are schematically listed as follows:

[0025] (1) Problem of frequent interface calls. Since the same data request may be initiated by multiple threads simultaneously, frequently calling the external system interface increases the pressure on the external system and causes the service response to slow down.

[0026] (2) Problem of resource contention. In high-concurrency scenarios, multiple threads will compete for access to resources, which easily leads to a decline in system performance and even problems such as resource exhaustion.

[0027] (3) Problem of data consistency. Multiple threads concurrently access the external system. Due to reasons such as data updates, the data obtained by different threads may be inconsistent, affecting the accuracy and reliability of subsequent business processing.

[0028] To improve the above problems, the embodiments of the present disclosure provide a data acquisition method, device, and program product, which will be elaborated in detail below.

[0029] Figure 1 As shown in the flowchart of a data acquisition method provided by an embodiment of the present disclosure, this method can be executed by a data acquisition device, where the device can be implemented by software and / or hardware, and is generally integrated in an electronic device. The electronic device can be a user device or a server, which is not limited herein. As Figure 1 shown, this method mainly includes the following steps S102 to S106:

[0030] Step S102, when receiving data acquisition requests initiated by multiple threads for target data in a target system, instruct the multiple threads to respectively request to obtain a distributed lock.

[0031] The target system can be a third-party system (also simply referred to as an external system). The embodiments of the present disclosure do not limit the target data. For example, the target data can be relevant data of a specified service, such as third-party data like customer credit investigation. It can be understood that in some cases, there may be multiple parallel policies that need to access the same data in the third-party system simultaneously. Different policies will separately call different threads to execute, and there will be a situation where different threads initiate data acquisition requests for the same data in the external system. At this time, problems such as frequent external system interface calls, resource contention, and difficulty in ensuring data consistency obtained by different threads will occur. To improve this problem, the embodiments of the present disclosure introduce a distributed lock mechanism. The distributed lock can effectively control the access to shared resources and ensure that only one thread can obtain the external system data at the same time, avoiding repeated calls and resource competition. Specifically, the distributed lock is a mechanism that can be used to coordinate the mutually exclusive access of multiple processes to shared resources in a distributed system. Exemplarily, the distributed lock can be implemented based on Zookeeper. Zookeeper is a distributed coordination tool that can provide relatively powerful lock management capabilities and provide a fair lock mechanism to ensure that the threads requesting the lock can obtain the lock in order, and only one thread can obtain the lock at the same time, thus avoiding problems such as priority inversion caused by resource contention. Moreover, Zookeeper can perceive the state change of the lock in real time through monitoring, which helps to simplify the management of concurrent requests. The embodiments of the present disclosure do not limit the specific implementation manner of the distributed lock, and can also be implemented by technologies or dedicated tools such as databases and key-value storage systems, which are not limited herein.

[0032] Taking the implementation of the distributed lock based on Zookeeper as an example, in some specific implementation examples, the electronic device for executing the above data acquisition method can be installed with a Zookeeper application program, and the acquisition of the distributed lock by multiple threads can be achieved by calling the interface of the Zookeeper application program. For example, multiple threads can send requests to Zookeeper to acquire the distributed lock and attempt to acquire the distributed lock. The specific manner in which the thread requests to acquire the distributed lock can refer to the related art and will not be elaborated herein.

[0033] Step S104, access the target system through the first thread among the multiple threads to obtain the target data in the target system, and cache the target data in a specified database; wherein, the first thread is the thread that successfully requests to acquire the distributed lock among the multiple threads.

[0034] Due to the mutual exclusion of the distributed lock, only one thread among multiple threads can successfully acquire the distributed lock, that is, the number of the first threads is one. For ease of understanding, taking the implementation of the distributed lock based on Zookeeper as an example, multiple threads respectively send requests to Zookeeper to acquire the distributed lock, and only the first thread that successfully requests to acquire the distributed lock is allowed to access the target system to obtain the target data in the target system. Further, in order to reduce the access frequency of the target system, the embodiments of the present disclosure may cause the first thread to cache the successfully acquired target data into a specified database so that other threads can directly obtain the target data from the specified database. The embodiments of the present disclosure do not limit the specific implementation manner of the specified database. Exemplarily, the specified database may be a Redis cache database.

[0035] Step S106, obtaining the target data from the specified database through a second thread among the multiple threads; wherein, the second thread is a thread that fails to request to acquire the distributed lock among the multiple threads. The second thread is also the thread other than the first thread among the multiple threads.

[0036] Since the first thread has stored the target data obtained from the target system in the specified database, the second thread that fails to acquire the distributed lock does not need to access the target system anymore, but can directly obtain the target data from the specified database. This can not only ensure the consistency of the data obtained by multiple threads, but also, compared with the related art where multiple threads simultaneously access the target system to compete for resources, the above method only requires one thread to access the target system, greatly reducing the access frequency of the external system and better alleviating problems such as frequent external system interface calls, resource contention, and data inconsistency.

[0037] In order to fully ensure that the second thread that fails to request to acquire the distributed lock can obtain the target data, in some embodiments, the above step S106, that is, the step of obtaining the target data from the specified database through the second thread among the multiple threads, may be executed with reference to the following steps (1) and (2):

[0038] Step (1), listening to the current state of the distributed lock; wherein, the current state includes the held state or the released state. In some specific implementation examples, the Zookeeper listening mechanism may be used to listen to the current state of the lock.

[0039] Step (2), in response to the current state being the release state, the second thread among multiple threads obtains target data from a specified database. In the embodiments of the present disclosure, the first thread usually releases the lock only after caching all the obtained target data in the specified database. Therefore, if it is monitored that the lock changes from the held state to the release state, it indicates that the target data is probably cached in the specified database. At this time, the second thread is then instructed to obtain the target data from the specified database, thus better ensuring that the second thread can accurately and completely obtain the target data from the specified database without having to access the target system again, reducing the frequency of repeated calls to external interfaces.

[0040] Furthermore, in order to ensure that the first thread can release the lock in a timely manner and avoid problems such as deadlocks or long-term resource occupation, in some embodiments, the embodiments of the present disclosure can release the distributed lock through the first thread in response to meeting a preset lock release condition.

[0041] Exemplarily, the lock release condition includes one or more of the following: all the target data has been cached in the specified database, the duration for which the first thread holds the distributed lock reaches a preset lock holding duration threshold, and it is monitored that the first thread is abnormal. In specific implementation, the first thread can be set to release the lock in a timely manner after caching the target data in the specified database to avoid resource occupation and provide an opportunity for other threads to obtain the lock earlier. Additionally, considering that external systems and networks are uncontrollable, situations such as target system exceptions and network exceptions may occur, resulting in the first thread being unable to access the target system normally and obtain the required target data in a timely manner, and thus unable to cache the target data in the specified database in a timely manner. Based on this, a lock holding duration threshold can be set to avoid the situation where the first thread holds the lock for a long time but cannot obtain and cache the target data in a timely manner, affecting other threads. By setting the lock holding duration threshold, the lock can be forced to be released after the first thread holds the lock for a long time, thereby giving other threads the opportunity to request and obtain the distributed lock again. Additionally, considering that the first thread may also be abnormal, the lock can be forced to be released when the first thread is abnormal, which can also avoid problems such as deadlocks or long-term resource occupation.

[0042] Further, the embodiments of the present disclosure provide a method for dynamically adjusting the lock holding duration threshold, so as to dynamically adjust the duration for which the first thread holds the distributed lock. Exemplarily, the embodiments of the present disclosure may determine the lock holding duration threshold based on the number of multiple threads. That is, the lock holding duration threshold may be dynamically adjusted based on the number of multiple threads. Exemplarily, the number of threads in concurrent requests may be monitored in real time. If the number of threads requesting the distributed lock simultaneously is large, the lock holding duration threshold may be appropriately extended. The embodiments of the present disclosure fully consider that the number of threads will affect the processing speed of tools for implementing distributed locks such as Zookeeper. For example, if the thread concurrency is large, the ability of tools for implementing distributed locks such as Zookeeper to process thread requests will be weakened, the processing speed will decrease, and the time taken for the thread to obtain data will also be relatively long. To ensure that thread requests can be processed normally as much as possible and avoid the first thread being forced to release the lock before caching the target data into the specified database due to a short lock holding duration threshold, the embodiments of the present disclosure may appropriately extend the lock holding duration threshold when the number of threads is large, so as to better ensure that the first thread that successfully obtains the lock caches all the target data into the specified database before releasing the lock. In some specific implementation examples, if the number of multiple threads is less than a preset number threshold, the lock holding duration threshold may be set to a first threshold. If the number of multiple threads is greater than the preset number threshold, the lock holding duration threshold may be set to a second threshold, where the second threshold is greater than the first threshold, and the second threshold may also be dynamically adjusted based on a preset algorithm, which can be flexibly set specifically and will not be limited here. By the above method, in high-concurrency request and high-load scenarios, by dynamically adjusting the lock holding duration threshold and correspondingly dynamically adjusting the lock holding duration, the success rate of the first thread successfully obtaining the target data and caching the target data into the specified database can be effectively improved, and then the second thread can also successfully obtain the target data from the specified database, thereby reducing frequent lock competition and improving the system concurrency processing performance.

[0043] The embodiments of the present disclosure may introduce a health check mechanism to detect whether the first thread is abnormal at a preset time interval, that is, regularly check the thread status of the lock, and forcibly release the lock of the first thread in a timely manner when it is detected that the first thread is abnormal, preventing the lock release from failing and avoiding problems such as deadlocks or long-term resource occupation.

[0044] To further reduce the time taken for the first thread to obtain and cache data, the foregoing step S104, that is, the step of accessing the target system through the first thread among multiple threads to obtain the target data in the target system, may be executed with reference to the following steps 1 to 3:

[0045] Step 1, query from a specified database through a first thread among multiple threads whether there is target data in a target system. That is, after the first thread successfully obtains a distributed lock, it can first query from the specified database whether there is the required data.

[0046] Step 2, in response to querying the first data in the target data from the specified database, obtain the first data from the specified database through the first thread, and access the target system to obtain the second data in the target data; wherein, the first data is at least part of the target data, and the second data is the data in the target data other than the first data. In some embodiments, if there is at least part of the required target data (the first data) in the specified database, the first thread can only obtain the remaining data (the second data) in the target data that is not cached in the specified database from the target system, and cache the remaining data in the specified database. If the specified database already stores the target data in advance, the foregoing first data is the target data, and at this time the second data is empty. In other words, the first thread does not need to obtain data from the target system anymore and can directly release the distributed lock.

[0047] Step 3, in response to not querying the target data from the specified database, access the target system through the first thread to obtain the target data in the target system.

[0048] Through the above method, if there is at least part of the data required by the first thread stored in the specified database in advance, the first thread does not need to obtain the data from the target system and cache the data in the specified database anymore. Therefore, the time taken for the first thread to obtain data from the target system and the time taken for data caching in the specified database can be effectively shortened.

[0049] In order to be able to shorten the time taken for a thread to obtain and cache data, in some embodiments, the embodiments of the present disclosure may also pre-execute the following steps a and / or step b:

[0050] Step a, in response to detecting that there is invalid data in the specified database, obtain valid data corresponding to the invalid data from the target system, and replace the invalid data in the specified database with the valid data.

[0051] In practical applications, the data in the specified database can all be marked with time information such as acquisition time and valid duration. Based on the time information marked for each data and the valid duration, it is possible to determine whether the data is valid currently. The data that has passed the valid duration is invalid data. To prevent subsequent threads from obtaining invalid data and needing to access the target system again, the embodiments of the present disclosure can pre-obtain the valid data corresponding to the invalid data from the target system, that is, obtain the data within the valid duration, and use it to replace the invalid data. For example, for credit investigation data, its valid duration is 30 days. Once the credit investigation data cached in the specified database exceeds 30 days, it can be regarded as invalid data and the latest credit investigation data needs to be obtained from the target system again. In practical applications, a listening mechanism can be used to regularly determine whether there is a situation where the cache has expired. If the cache has expired (that is, invalid data is detected), a cache refresh operation is actively triggered to ensure that the thread can directly obtain valid data from the specified database, making the cached valid data as consistent as possible with the data in the target system, and also enabling the first thread to directly obtain the required data from the specified database in advance, reducing the amount of data obtained by the first thread from the target system, and shortening the time taken for the first thread to cache data in the specified database, facilitating the first thread to release the lock in the shortest possible time.

[0052] Step b, in response to receiving a pre-acquisition instruction for standby data, obtain the standby data from the target system and pre-store the standby data in the specified database.

[0053] To reduce lock contention and external system calls under high-frequency requests, the embodiments of the present disclosure can pre-batch obtain the data that may be requested from the target system and cache it in the specified database. For example, assume that Service A and Service B need to be executed sequentially. When executing Service A, it is highly likely that Service B will be executed subsequently. Therefore, even if there is no thread currently requesting the data of Service B, it is still possible to choose to pre-obtain the data of Service B during the idle period, initiate a pre-acquisition instruction for the data of Service B, and pre-cache the data of Service B as standby data in the specified database. When a subsequent thread requests to obtain the data of Service B, it can directly obtain it from the specified database without having to access the target system again, or only need to obtain the data of Service B that has not been cached from the target system, which can greatly optimize the system efficiency.

[0054] Furthermore, considering that there may be situations such as network anomalies and specified database anomalies, resulting in the second thread being unable to obtain the target data from the specified database, the method provided by the embodiments of the present disclosure further includes: in response to the second thread not successfully obtaining the target data from the specified database, instruct the second thread to request the distributed lock again, so as to ensure that the second thread can successfully obtain the target data for subsequent processing.

[0055] For ease of understanding, taking the implementation of the distributed lock based on Zookeeper as an example, at this time, the distributed lock can be simply referred to as the Zookeeper lock. The embodiments of the present disclosure further provide Figure 2 A schematic diagram of a data acquisition process shown in the figure shows that threads A to D simultaneously request to query the third-party data (corresponding to the aforementioned target data) in the external system. In specific implementation, all four threads need to request to acquire the lock. For each thread, it is necessary to determine whether it has successfully acquired the lock. For the thread that has successfully acquired the lock (assumed to be thread A), it can directly call the external interface to obtain the required third-party data, and write the third-party data into the Redis cache (corresponding to the aforementioned specified database), and then the lock can be released. For the threads that have failed to acquire the lock (assumed to be threads B to D), they need to monitor the status of the lock, and then acquire the third-party data in the Redis cache when it is confirmed that the lock is in the released state. Subsequently, it can be further determined whether the third-party data has been successfully acquired from the Redis. If so, the subsequent data processing tasks can be executed. If not, the lock can be requested again. It should be noted that each thread can execute its own data processing tasks based on the acquired third-party data, and the data processing tasks corresponding to different threads can be the same or different. The embodiments of the present disclosure do not limit this. The above specific steps can refer to the aforementioned related technologies and will not be elaborated here.

[0056] In summary, the data acquisition method provided by the embodiments of the present disclosure can effectively solve the problem of repeated calls when multiple threads concurrently access the external system interface by skillfully leveraging the distributed lock mechanism when multiple threads simultaneously acquire the same data in the external system, and has at least the following advantages:

[0057] (1) Fairness. Specifically, the distributed lock mechanism has fairness, that is, it is allocated in the order of requesting the lock, which can ensure that each thread in this system (the system for executing the data acquisition method provided by the embodiments of the present disclosure) has the opportunity to acquire the lock fairly, avoiding the problem of resource skew caused by lock competition.

[0058] (2) Reducing the interface call frequency, avoiding resource contention among multiple threads, and effectively ensuring the consistency of the data obtained by multiple threads. Specifically, only the thread that has successfully acquired the lock can call the external interface to obtain the required data and cache the required data. Other threads do not need to call the external interface anymore, but can directly obtain the cached data, which can effectively avoid many problems caused by multiple threads accessing the external system.

[0059] (3) It has efficient concurrent processing capabilities. Specifically, it can monitor the number of threads of concurrent requests in real time and dynamically adjust the holding duration threshold of the distributed lock. For example, in high-concurrency request and high-load scenarios, it can dynamically extend the lock holding time to reduce frequent lock contention, thereby improving the system's concurrent processing performance. Moreover, through the listening mechanism, the state of the lock can be monitored in a timely manner, and the thread can perceive the state change at the first moment when the lock is released and quickly switch to obtaining data from the cache, avoiding the system overhead and performance loss caused by polling. In addition, through the cache prefetch mechanism, the frequent lock contention and the number of external system calls in a high-concurrency environment can be effectively reduced. For example, when the cache expires, the cache refresh operation is actively triggered, and the data of the external system that may be requested is pre-obtained and cached to ensure the consistency between the cache data and the external system as much as possible, and to reduce the time-consuming for the thread to obtain data from the external system and cache the data. The thread can directly obtain the cache data, reducing the lock contention and external system calls under high-frequency requests, further optimizing the system efficiency and improving the overall throughput and concurrent processing capabilities of the system.

[0060] (4) It has high availability. Specifically, the system has a fault tolerance mechanism. In many cases such as lock acquisition failure, call timeout, cache invalidation, and thread exception, it can automatically recover and provide alternative solutions. Moreover, when the thread holding the lock encounters an exception or is unresponsive for a long time, the system automatically detects and releases the lock through the health check mechanism, effectively preventing deadlock and thread blocking problems, comprehensively ensuring the high availability of the system. Further, in some implementation examples, the distributed lock can be implemented based on Zookeeper. Since Zookeeper has strong distributed coordination capabilities, multiple nodes of Zookeeper can share the lock state, ensuring the consistency of concurrent operations even across nodes. The high availability of Zookeeper can also effectively ensure the high availability of this system.

[0061] Corresponding to the foregoing data acquisition method, an embodiment of the present disclosure further provides a data acquisition device. Figure 3 It is a schematic structural diagram of a data acquisition device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and is generally integrated in an electronic device, such as Figure 3 As shown, the data acquisition device includes:

[0062] A lock acquisition module 302, configured to, when receiving data acquisition requests initiated by multiple threads for target data in a target system, cause the multiple threads to respectively request to acquire a distributed lock;

[0063] The first acquisition module 304 is configured to access the target system through the first thread among the multiple threads to acquire target data in the target system and cache the target data into a specified database; wherein, the first thread is the thread that successfully requests to acquire a distributed lock among the multiple threads;

[0064] The second acquisition module 306 is configured to acquire the target data from the specified database through the second thread among the multiple threads; wherein, the second thread is the thread that fails to request to acquire a distributed lock among the multiple threads.

[0065] The above device can not only effectively ensure that multiple threads can all acquire data, but also effectively improve the situation where multiple threads all access the target system to acquire data, thereby better alleviating problems such as frequent external system interface calls and resource contention caused thereby.

[0066] In some embodiments, the second acquisition module 306: monitors the current state of the distributed lock; wherein, the current state includes a held state or a released state; in response to the current state being the released state, acquires the target data from the specified database through the second thread among the multiple threads.

[0067] In some embodiments, the device further includes a lock release module, configured to release the distributed lock through the first thread in response to meeting a preset lock release condition.

[0068] In some embodiments, the lock release condition includes one or more of the following: all of the target data has been cached into the specified database, the duration for which the first thread holds the distributed lock reaches a preset lock holding duration threshold, it is detected that the first thread is abnormal.

[0069] In some embodiments, the device further includes a threshold determination module, configured to determine the lock holding duration threshold based on the number of the multiple threads.

[0070] In some embodiments, the device further includes an abnormality detection module, configured to detect whether the first thread is abnormal at a preset time interval.

[0071] In some embodiments, the device further includes a data processing module, configured to, in response to detecting invalid data in the specified database, acquire valid data corresponding to the invalid data from the target system and replace the invalid data in the specified database with the valid data; and / or, in response to receiving a pre-acquisition instruction for backup data, acquire the backup data from the target system and pre-store the backup data into the specified database.

[0072] In some embodiments, the first acquisition module 304 is specifically configured to: query from the specified database through a first thread among the multiple threads whether the target data in the target system is cached; in response to querying the first data in the target data from the specified database, obtain the first data from the specified database through the first thread, and access the target system to obtain the second data in the target data from the target system; wherein, the first data is at least part of the target data, and the second data is the data in the target data other than the first data; in response to not querying the target data from the specified database, access the target system through the first thread to obtain the target data in the target system.

[0073] In some embodiments, the apparatus further includes a re-request module, configured to, in response to the second thread not successfully obtaining the target data from the specified database, cause the second thread to request to obtain the distributed lock again.

[0074] The data acquisition apparatus provided by the embodiments of the present disclosure can execute the data acquisition method provided by any embodiment of the present disclosure, and has functional modules and beneficial effects corresponding to the execution of the method.

[0075] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device embodiments described above can refer to the corresponding process in the method embodiments, and will not be elaborated here.

[0076] The embodiments of the present disclosure provide an electronic device, which includes: a storage device on which a computer program is stored; a processing device configured to execute the computer program in the storage device to implement the steps of any one of the methods in the present disclosure.

[0077] Next, refer to Figure 4 , which shows a schematic structural diagram of an electronic device 400 suitable for implementing the embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Tablet Computers), PMPs (Portable Multimedia Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The electronic device shown is only an example, and should not bring any limitation to the functions and usage scopes of the embodiments of the present disclosure.

[0078] As Figure 4As shown, the electronic device 400 may include a processing device (such as a central processing unit, a graphics processing unit, etc.) 401, which may perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0079] Generally, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or wiredly to exchange data. Although Figure 4 an electronic device 400 with various devices is shown, it should be understood that it is not required to implement or have all the shown devices. Instead, more or fewer devices may be implemented or had.

[0080] Specifically, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, an embodiment of the present disclosure includes a computer program carried on a non-transitory computer-readable medium, the computer program including program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

[0081] In addition to the above methods and devices, embodiments of the present disclosure may also be computer program products, which include computer program instructions that, when run on a processor, cause the processor to execute the methods provided by the embodiments of the present disclosure. The computer program products may be written in any combination of one or more programming languages for programming code to perform the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0082] In addition, embodiments of the present disclosure may also be computer-readable storage media, on which computer program instructions are stored, and the computer program instructions, when run on a processor, cause the processor to execute the data acquisition method provided by the embodiments of the present disclosure.

[0083] The computer-readable storage media may adopt any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples (non-exhaustive list) of the readable storage medium include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0084] Embodiments of the present disclosure also provide a computer program product, including computer programs / instructions, which, when executed by a processor, implement the data acquisition method in the embodiments of the present disclosure.

[0085] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the users and the users' authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0086] For example, when responding to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application, a server, or a storage medium that performs the operations of the present disclosure's technical solution based on the prompt message.

[0087] As an optional but non-limiting implementation manner, when responding to receiving an active request from a user, the manner of sending a prompt message to the user can be, for example, in the form of a pop-up window. The prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0088] It can be understood that the above notification and obtaining user authorization process is only illustrative and does not constitute a limitation on the implementation manner of the present disclosure. Other manners that meet relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0089] It should be noted that in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0090] The above are only specific implementation manners of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data acquisition method, characterized in that: include: In the case of receiving data acquisition requests initiated by multiple threads for target data in the target system, allowing the multiple threads to request to acquire distributed locks respectively; Accessing the target system through a first thread among the multiple threads to obtain target data in the target system, and caching the target data in a designated database; wherein the first thread is a thread among the multiple threads that successfully requests to obtain a distributed lock; The target data is obtained from the designated database through a second thread among the multiple threads; wherein the second thread is a thread among the multiple threads that fails in requesting to obtain a distributed lock.

2. The method according to claim 1, characterized in that The acquiring the target data from the designated database through a second thread among the multiple threads includes: Monitoring the current state of the distributed lock; wherein the current state includes a holding state or a releasing state; In response to the current state being a released state, the target data is acquired from the designated database through a second thread among the multiple threads.

3. The method according to claim 1, characterized in that The method further comprises: In response to satisfying a preset lock release condition, releasing the distributed lock through the first thread.

4. The method according to claim 3, characterized in that The lock release condition includes one or more of the following: the target data has been cached in the designated database, the length of time the first thread holds the distributed lock reaches a preset lock holding time threshold, and an abnormality of the first thread is detected.

5. The method according to claim 4, characterized in that The method further comprises: The lock holding time threshold is determined based on the number of the multiple threads.

6. The method according to claim 1, characterized in that The method further comprises: In response to detecting that there is invalid data in the designated database, obtaining valid data corresponding to the invalid data from the target system, and replacing the invalid data in the designated database with the valid data; and / or, In response to receiving a pre-acquisition instruction for standby data, the standby data is acquired from the target system, and the standby data is pre-stored in the designated database.

7. The method according to claim 1 or 6, characterized in that: The accessing the target system through the first thread among the multiple threads to obtain target data in the target system includes: querying, through a first thread among the multiple threads, from the designated database whether target data in the target system is cached; In response to querying the first data in the target data from the designated database, obtaining the first data from the designated database through the first thread, and accessing the target system to obtain second data in the target data from the target system; wherein the first data is at least part of the data in the target data, and the second data is data in the target data other than the first data; In response to the target data not being found in the query from the designated database, the target system is accessed through the first thread to obtain the target data in the target system.

8. The method according to claim 1, characterized in that The method further comprises: In response to the second thread failing to successfully obtain the target data from the designated database, the second thread is instructed to request to obtain the distributed lock again.

9. An electronic device, characterized in that: The electronic device comprises: a storage device having a computer program stored thereon; A processing device, used to execute the computer program in the storage device to implement the steps of the data acquisition method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the data acquisition method according to any one of claims 1 to 8 when executed by a processor.

Images