High concurrency data storage method and device

一种数据存储、数据的技术，应用在数据处理领域，能够解决数据库链接多、服务器宕机、占用资源多等问题，达到提升存储效率、避免宕机的效果

Active Publication Date: 2015-05-06
BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD +1
View PDF5 Cites 27 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, for some old business systems, the cost of replacing new storage methods will be relatively high, so traditional relational databases are still used to store data, such as MySql, Oracle, etc.
When the system is faced with high-concurrency data storage, it will lead to too many links to the database, occupying too many resources, reducing storage efficiency, and causing server downtime and other problems

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • High concurrency data storage method and device
  • High concurrency data storage method and device
  • High concurrency data storage method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0026] figure 1 It is a flow chart of the high-concurrency data storage method provided by Embodiment 1 of the present invention. This embodiment is applicable to the storage of high-concurrency data. The method can be executed by the server, and specifically includes the following steps:

[0027] Step 110, receiving high-concurrency data sent by multiple clients.

[0028] Among them, high concurrency refers to a relatively large number of visits at a certain moment. The server receives requests from multiple clients, and many clients request at the same time to form highly concurrent data.

[0029] Step 120, push the high-concurrency data to the main data queue, and respond to the corresponding client.

[0030] Among them, the data queue adopts distributed MQ (Message Queue, message queue), which can support distributed expansion, and can also make the framework highly available, and still have relatively objective performance when processing big data. Among them, ActiveMQ...

Embodiment 2

[0054] figure 2 It is an architectural design diagram of the high-concurrency data storage method provided by Embodiment 2 of the present invention. Such as figure 2 As shown, the main data queue is the main entry queue for all data. When the data is pushed to the queue, the result will be returned immediately to improve the response speed; the thread pool (Thread Pool) provides multi-threaded data processing and pushes the data to the corresponding The local queue can improve the consumption speed of messages, and at the same time, it can also use the characteristics of MQ to persist data and prevent data loss; the local queue is used to store data after fragmentation and prevent data loss after fragmentation; work (Work )1, Work 2, and Work 3 can independently control the amount of captured data and the time interval for capturing data by timing / instantly consuming data, processing the data and storing it in the database.

[0055] image 3 It is a flow chart of the high...

Embodiment 3

[0067] Figure 4 is a schematic diagram of a high-concurrency data storage device provided in Embodiment 3 of the present invention. The high-concurrency data storage device provided in this embodiment is used to implement the high-concurrency data storage method provided in Embodiment 1. Such as Figure 4 As shown, the high-concurrency data storage device provided in this embodiment includes: a receiving module 410 , a first push module 420 , a consumption module 430 , a fragmentation module 440 , a second push module 450 and a storage module 460 .

[0068] Among them, the receiving module 410 is used to receive high concurrency data sent by multiple clients; the first push module 420 is used to push the high concurrency data to the main data queue and respond to corresponding clients; the consumption module 430 is used to Utilize multithreading to consume the high concurrency data in the main data queue; the fragmentation module 440 is used to fragment the high concurrency...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a high concurrency data storage method and device. The method comprises the steps; receiving high concurrency data transmitted by a plurality of clients; pushing the high concurrency data to a main data queue, and responding to the corresponding client; consuming the high concurrency data in the main data queue by utilizing a multithread; fragmenting the high concurrency data according to the quantity of a local queue; pushing the fragmented high concurrency data to the local queue; consuming the high concurrency data in the local queue, and storing the high concurrency data into a database. The high concurrency data is cashed by utilizing the main data queue and the local queue, the data storage is carried out by adopting an asynchronous storage way, the pressure of the database is alleviated, the downtime problem of the database under the high concurrency storage situation can be avoided, and the storage efficiency of the high concurrency data can be improved.

Description

technical field [0001] Embodiments of the present invention relate to data processing technologies, and in particular to a high-concurrency data storage method and device. Background technique [0002] With the explosive growth of Internet information, more and more Internet servers will encounter the environment of high concurrency and massive data, and in this environment, they will face the problem of high concurrency data storage. In order to solve this problem, some NoSql (Not Only SQL, non-relational database) languages ​​such as Hadoop, MongoDB, and Hbase have been widely used. [0003] However, for some old business systems, the cost of replacing new storage methods will be relatively high, so traditional relational databases are still used to store data, such as MySql, Oracle, etc. When the system is faced with high-concurrency data storage, it will lead to too many links to the database, occupying too many resources, reducing storage efficiency, and causing proble...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30G06F9/46
CPCG06F16/22G06F16/278G06F16/2308G06F9/546G06F9/4881G06F16/2471G06F16/24578G06F9/45504G06F9/461G06F9/485
Inventor 兰健
Owner BEIJING JINGDONG SHANGKE INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap