Method and device for realizing mass data access, medium and equipment
Through the combination of Bloom filter and cache database, the problem of inefficiency of relational databases in massive data processing is solved, the stability and efficiency of the system are achieved, and the risk of downtime is avoided.
Patent Information
- Application Number
- CN202411435744.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing relational databases are inefficient when processing massive data, resulting in insufficient system stability, which may lead to downtime or crashes, and cannot meet the needs of fast data acquisition and analysis.
Use the Blonde filter and cache database to query the data in the Blonde filter by obtaining the hash value of the customer access request. If it exists, it will be returned from the cache database. If it does not exist, it will be called and a random database is set in the cache database. In the failure time, establish a distribution lock mechanism and cache database cluster to realize automatic switching and self-rescue survival mechanism.
It reduces the pressure on relational databases, ensures the stability and operation efficiency of the system in high concurrency situations, and avoids the risk of downtime.
Smart Images

Figure CN118939742A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method, device, medium and equipment for realizing mass data access, belonging to the technical field of data processing. Background Art
[0002] With the rise of the Internet and the improvement of informatization, the timeliness of data collection is required to be faster and faster, and the stability of the system is required to be higher and higher. At the same time, in order to cope with the upload of massive data, the system needs to collect data in batches, clean data, store data and conduct subsequent data analysis. However, in the face of massive data processing, relational databases have become a bottleneck in system operation. The low efficiency in storage and query has brought passivity to R&D personnel. In extreme cases, the database may directly cause downtime or collapse due to pressure problems. Summary of the invention
[0003] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a method, device, medium and equipment for realizing mass data access.
[0004] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.
[0005] In one aspect, the present invention discloses a method for implementing mass data access, comprising: Obtain the customer access request, and query the hash value corresponding to the requested data in the Bloom filter according to the customer access request; If the corresponding hash value is found in the Bloom filter, the requested data is queried from the pre-built cache database according to the corresponding hash value. If the requested data is found, it is returned to the client. If the requested data is not found, the relational database is called to obtain the requested data from the relational database and the requested data is rewritten into the cache database. The data in the cache database is set with a random expiration time. If the data is not accessed within the random expiration time, the data is deleted from the cache database. If the data is accessed within the random expiration time, the random expiration time of the data is reset. If the corresponding hash value is not found in the Bloom filter, the returned data does not exist.
[0006] Furthermore, the construction of the cache database includes: Collecting massive data uploaded by the client, cleaning and sorting the massive data to obtain the required valuable data, and storing it in a relational database. The required valuable data is the preset data that meets the target requirements; The key value of the required valuable data is written into the cache database, and the hash value of the data is added to the Bloom filter. At the same time, a random expiration time is set for each piece of data to complete the construction of the cache database.
[0007] Furthermore, it also includes: activating a distributed lock mechanism when receiving a request to modify data in a relational database, wherein the distributed lock mechanism is used to limit only one request to modify the data at the same time, and after modifying the data in the relational database, synchronously modifying the data in the cache database.
[0008] Furthermore, it also includes: establishing a cache database cluster, wherein the cache database cluster includes a plurality of cache databases; The multiple cache databases are in a one-write-many-read relationship, and cache services are provided between the multiple cache databases through a heartbeat packet mechanism. When an abnormality is detected in the cache database currently providing services, the heartbeat packet mechanism is used to automatically switch to another cache database for cache services.
[0009] Furthermore, it also includes: a self-rescue and survival mechanism, which is used to shield or reduce the frequency of corresponding data services from low to high according to preset data priorities when it is detected that the customer access request reaches a preset peak within a preset time.
[0010] In a second aspect, the present invention further discloses a device for implementing mass data access, comprising: The acquisition module is used to obtain the customer access request and query the hash value corresponding to the requested data in the Bloom filter according to the customer access request; A processing module is used to: if the corresponding hash value is queried in the Bloom filter, query the requested data from the pre-built cache database according to the corresponding hash value; if the requested data is queried, return it to the client; if the requested data is not queried, call the relational database, obtain the requested data from the relational database, and rewrite the requested data into the cache database; the data in the cache database is set with a random expiration time, if the data is not accessed within the random expiration time, the data is deleted from the cache database, and if the data is accessed within the random expiration time, the random expiration time of the data is reset; If the corresponding hash value is not found in the Bloom filter, the returned data does not exist.
[0011] In a third aspect, the present invention further discloses a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, enable the computing device to execute the method of the first aspect.
[0012] In a fourth aspect, the present invention further discloses a computer device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method of the first aspect.
[0013] The beneficial effects achieved by the present invention are: The present invention is based on a Bloom filter and a cache database, can reduce the pressure on a relational database, and ensures the stability of system operation in the face of high concurrency and processing of massive data. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Figure 1 It is a schematic diagram of the construction process of the cache database of the present invention; Figure 2 It is a schematic diagram of the process of reading data of the present invention. DETAILED DESCRIPTION
[0015] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.
[0016] In the description of the present invention, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, features defined as "first", "second", and the like may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.
[0017] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood by specific circumstances.
[0018] Embodiment 1: This embodiment introduces a method for implementing massive data access, including: Obtain the customer access request, and query the hash value corresponding to the requested data in the Bloom filter according to the customer access request; If the corresponding hash value is found in the Bloom filter, the requested data is queried from the pre-built cache database according to the corresponding hash value. If the requested data is found, it is returned to the client. If the requested data is not found, the relational database is called to obtain the requested data from the relational database and the requested data is rewritten into the cache database. The data in the cache database is set with a random expiration time. If the data is not accessed within the random expiration time, the data is deleted from the cache database. If the data is accessed within the random expiration time, the random expiration time of the data is reset. If the corresponding hash value is not found in the Bloom filter, the returned data does not exist.
[0019] like Figure 1 As shown, the construction of the cache database includes: Collect massive data uploaded by the client, clean and organize the massive data to obtain the required valuable data, and store it in a relational database. The required valuable data is the preset data that meets the target requirements (such as data resources that bring economic benefits to the company or other required data); The key value of the required valuable data is written into the cache database, and the hash value of the data is added to the Bloom filter. At the same time, a random expiration time is set for each piece of data to complete the construction of the cache database.
[0020] It also includes: activating a distributed lock mechanism when receiving a request to modify data in a relational database, wherein the distributed lock mechanism is used to limit only one request to modify the data at the same time, and after modifying the data in the relational database, synchronously modifying the data in the cache database.
[0021] It also includes: establishing a cache database cluster, wherein the cache database cluster includes a plurality of cache databases; The multiple cache databases are in a one-write-many-read relationship, and cache services are provided between the multiple cache databases through a heartbeat packet mechanism. When an abnormality is detected in the cache database currently providing services, the heartbeat packet mechanism is used to automatically switch to another cache database for cache services.
[0022] It also includes: a self-rescue and survival mechanism, which is used to shield or reduce the frequency of corresponding data services from low to high according to pre-set data priorities when it is detected that a short-term customer access request reaches a preset peak value.
[0023] Embodiment 2, as Figure 2 As shown, this embodiment introduces a method for implementing massive data access, and the specific steps are: (1) Collect client upstream data in batches, store it in a relational database, and synchronize it to the REDIS cache database at the same time; (2) Add a REDIS cache database to the existing architecture to handle concurrent queries from a large number of users; (3) REDIS cache database supports key / value, List, Set, Zset, hash and other storage methods; (4) Reasonably set the size and expiration time of local cache and distributed cache to ensure data consistency and timeliness; (5) Make data persistent to ensure that it can be restored after the system restarts, reducing the pressure on the database; (6) Establish a backup mechanism so that when a problem occurs with the cache server, it can be quickly restored or automatically switched to the backup system; (7) Hotspot data processing: monitoring and identifying data with high access frequency, and paying special attention to these data; (8) Establish a distributed lock mechanism. When updating the data of a hot data key, use the distributed lock mechanism to ensure that only one request can be made to build the cache reconstruction of this hot data at the same time, to avoid a large number of requests hitting the database at the same time and causing server interruption; (9) At the same time, message queues are used to buffer and serialize requests for high-frequency hot data; (10) Emergency survival mechanism: When cache avalanche or other system abnormalities occur, non-core functions are temporarily shut down to ensure the normal operation of core functions. At the same time, through request flow limiting, algorithms are used to limit access frequency and other mechanisms to ensure that the system is within the tolerance range. In high-traffic situations, priorities are set to prioritize the processing of important customers or requests.
[0024] The specific working principle is: After the server gateway interacts with the client to collect massive amounts of data in batches, the server gateway cleans and organizes the data to obtain the required valuable data information (at the same time, through the collection, cleaning, organization, and analysis of massive amounts of data, it can provide the management with graphical analysis report data to improve effective management measures), and stores it in the relational database. At the same time, the data key value is written into the cache database, and the hash value of the data is added to the Bloom filter. At the same time, a random expiration time is set for each piece of data to ensure that the data will not become invalid in large quantities at a certain moment, causing the cache database query to fail, causing a large number of requests to directly access the relational database, causing excessive pressure on the relational database, and the risk of server downtime.
[0025] When a large number of users access, they are first filtered through the cache Bloom filter. When querying the Bloom filter, if the returned data does not exist, it proves that the relational database does not exist. If the returned data exists, it does not necessarily mean that the data exists in the cache database. The Bloom filter is used to prevent the breakdown and avalanche of the cache database. If the data exists, the cache database is queried. After the data is queried, the random expiration time of the data is reset, indicating that the data is active data at this stage. For data that has not been queried for a long time, once the set time expires, it will be set as invalid and cleared from the cache database, while the data in the relational database will remain unchanged. If a user accesses the data, the relational database will be called to access the data only when the data is not found in the cache database, and the data will be rewritten into the cache database.
[0026] In order to prevent the relational database from being overwhelmed when a large number of users access NULL value data, a key value is set for accessing NULL value data and put it into the cache database. Subsequent users will directly return data from the cache database without querying the relational database.
[0027] The relational database update mechanism directly writes new data into the relational database after cleaning and sorting, and simultaneously writes it into the Bloom filter and cache database, and sets a random expiration time for each piece of data; for data that needs to be modified, a distributed lock mechanism is established to limit only one request to modify the data at the same time. After modifying the data in the relational database, the data in the cache database is modified synchronously, and other request modification operations are discarded.
[0028] Establish a cache database cluster. When a problem occurs in a cache database, the system automatically switches to other cache database servers without the user noticing and continues to provide services to the user, avoiding long-term downtime and causing a bad experience for the user.
[0029] A self-rescue and survival mechanism is established to automatically close access to non-core data when there are high concurrent requests to ensure the normal operation of core data, and to establish algorithm mechanisms and priorities to ensure that the system can provide services to users within an acceptable range.
[0030] After readjusting the system architecture in this way, the efficiency in processing massive amounts of data has been substantially improved, and the server no longer becomes unresponsive.
[0031] Embodiment 3 is based on the same inventive concept as Embodiment 1. This embodiment introduces a device for implementing massive data access, including: The acquisition module is used to obtain the customer access request and query the hash value corresponding to the requested data in the Bloom filter according to the customer access request; A processing module is used to: if the corresponding hash value is queried in the Bloom filter, query the requested data from the pre-built cache database according to the corresponding hash value; if the requested data is queried, return it to the client; if the requested data is not queried, call the relational database, obtain the requested data from the relational database, and rewrite the requested data into the cache database; the data in the cache database is set with a random expiration time, if the data is not accessed within the random expiration time, the data is deleted from the cache database, and if the data is accessed within the random expiration time, the random expiration time of the data is reset; If the corresponding hash value is not found in the Bloom filter, the returned data does not exist.
[0032] Embodiment 4 is based on the same inventive concept as Embodiment 1. This embodiment introduces a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by a computing device, the computing device executes the method of Embodiment 1.
[0033] Embodiment 5 is based on the same inventive concept as Embodiment 1. This embodiment introduces a computer device, including: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing the method described in Example 1.
[0034] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0035] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0036] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0037] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0038] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for implementing mass data access, characterized in that: include: Obtain the customer access request, and query the hash value corresponding to the requested data in the Bloom filter according to the customer access request; If the corresponding hash value is found in the Bloom filter, the requested data is queried from the pre-built cache database according to the corresponding hash value. If the requested data is found, it is returned to the client. If the requested data is not found, the relational database is called to obtain the requested data from the relational database and the requested data is rewritten into the cache database. The data in the cache database is set with a random expiration time. If the data is not accessed within the random expiration time, the data is deleted from the cache database. If the data is accessed within the random expiration time, the random expiration time of the data is reset. If the corresponding hash value is not found in the Bloom filter, the returned data does not exist.
2. The method for implementing mass data access according to claim 1, characterized in that: The construction of the cache database includes: Collecting massive data uploaded by the client, cleaning and sorting the massive data to obtain the required valuable data, and storing it in a relational database. The required valuable data is the preset data that meets the target requirements; The key value of the required valuable data is written into the cache database, and the hash value of the data is added to the Bloom filter. At the same time, a random expiration time is set for each piece of data to complete the construction of the cache database.
3. The method for implementing mass data access according to claim 2, characterized in that: Also includes: When a request to modify data in a relational database is received, a distributed lock mechanism is activated. The distributed lock mechanism is used to limit only one request to modify the data at the same time. After the data in the relational database is modified, the data in the cache database is modified synchronously.
4. The method for implementing mass data access according to claim 2, characterized in that: It also includes: establishing a cache database cluster, wherein the cache database cluster includes a plurality of cache databases; The multiple cache databases are in a one-write-many-read relationship, and cache services are provided between the multiple cache databases through a heartbeat packet mechanism. When an abnormality is detected in the cache database currently providing services, the heartbeat packet mechanism is used to automatically switch to another cache database for cache services.
5. The method for implementing mass data access according to claim 2, characterized in that: Also includes: A self-rescue and survival mechanism is used to shield or reduce the frequency of corresponding data services from low to high according to preset data priorities when it is detected that customer access requests reach a preset peak within a preset time.
6. A device for implementing mass data access, characterized in that: include: The acquisition module is used to obtain the customer access request and query the hash value corresponding to the requested data in the Bloom filter according to the customer access request; A processing module is used to: if the corresponding hash value is queried in the Bloom filter, query the requested data from the pre-built cache database according to the corresponding hash value; if the requested data is queried, return it to the client; if the requested data is not queried, call the relational database, obtain the requested data from the relational database, and rewrite the requested data into the cache database; the data in the cache database is set with a random expiration time, if the data is not accessed within the random expiration time, the data is deleted from the cache database, and if the data is accessed within the random expiration time, the random expiration time of the data is reset; If the corresponding hash value is not found in the Bloom filter, the returned data does not exist.
7. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions which, when executed by a computing device, cause the computing device to perform the method of any one of claims 1 to 5.
8. A computer device, characterized in that: include, One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods described in claims 1 to 5.
Citation Information
Patent Citations
Medicine database access method and device and computer equipment
CN112035479A
Caching method and device, electronic equipment and computer readable storage medium
CN112035529A
Query request filtering method applied to Redis client and Redis client
CN112818019A
Storage system, method and device for read receipts and readable storage medium
CN116595014A
Updating cached database query results
US20160171008A1
