Data storage method, system, device and medium

A data storage and message data technology, applied in the field of data processing, can solve problems such as data storage inconsistency, and achieve the effect of ensuring consistency and ensuring consistency

Inactive Publication Date: 2019-03-22
RUN TECH CO LTD BEIJING
View PDF9 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] Embodiments of the present invention provide a data storage method, system, device, and medium to solve the problem of inconsistency in data storage when the same data is stored in different slave machines under the big data framework

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data storage method, system, device and medium
  • Data storage method, system, device and medium
  • Data storage method, system, device and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0027] figure 1 It is a flow chart of the steps of a data storage method provided in the first embodiment of the present invention. This embodiment is applicable to the case of data storage under a big data processing framework, where the big data processing framework includes a host and at least one From the machine, the method can be executed by a data storage system, which can be implemented by software and / or hardware, and is generally integrated on the host of the big data processing framework, specifically including the following steps:

[0028] Step 110: Obtain message data generated by the business process, and store the message data in at least one target slave.

[0029] Among them, the message data can be a collection of various data generated in the business process, and can include structured data, semi-structured data, and unstructured data. For example, the message data can include consumption records, browsing records, browsing times, and Messages, etc.; the target s...

Embodiment 2

[0043] figure 2 It is a flow chart of the steps of the data storage method provided in the second embodiment of the present invention. This embodiment is an optional solution proposed on the basis of the foregoing embodiment. See figure 2 , The data storage method provided by the embodiment of the present invention includes:

[0044] Step 210: Obtain the message data generated by the business process, and store the message data in at least one target slave.

[0045] Step 220: When a data storage failure message fed back by at least one target slave is received, the message data and related failure information are stored as exception information in a preset exception list.

[0046] Step 230: Return to perform the data storage operation until the storage end condition is reached, and determine the error time of the corresponding message data according to the related failure information of the message data in each of the abnormal information.

[0047] Among them, the failure informati...

Embodiment 3

[0076] image 3 It is a schematic structural diagram of a data storage system provided in the third embodiment of the present invention. The system is based on the framework of big data storage and includes a master and at least one slave. The data storage system provided in the embodiment of the present invention can perform any of the present invention. The data storage method provided by the embodiment has functional modules and beneficial effects corresponding to the execution method, such as image 3 As shown, the system includes: a message data storage module 310, an exception list generation module 320, a directory log generation module 330, and a message data re-storage module 340.

[0077] Wherein, the message data storage module 310 is used to obtain message data generated by the business process, and store the message data in at least one target slave.

[0078] The abnormal list generating module 320 is configured to store the message data and related failure information ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data storage method, a system, a device and a medium. Acquiring message data generated by the business process and storing the message data in the target slave; When receiving the target slave feedback data storage failure message, the message data and the related failure information are stored in the exception list as the exception information. Returns to execute the data storage operation until the storage end condition is reached, and writes all the exception information contained in the exception list to the directory log; The target exception information in the directory log is extracted, the message data in the target exception information is processed, and the processed message data is restored based on the failure information of the message data. The method provided by the embodiment of the invention realizes that the storage result of each message data is monitored when the batch data is stored, and when the message data storage fails, the erroneous message data is stored again, so as to ensure the consistency of the message data stored by each slave when the same message data is stored by different slave computers.

Description

Technical field [0001] The embodiments of the present invention relate to the technical field of data processing, and in particular to a data storage method, system, device, and medium. Background technique [0002] Among the popular big data processing frameworks today, Spark is the most widely used. When the Spark application writes big data to multiple storage terminals such as redis, elasticsearch, hbase, and hdfs, big data storage will be generated due to big data write errors The problem of inconsistency. [0003] In order to ensure the consistent storage of big data under the Spark framework, there are currently only two common processing mechanisms. One is to use Kafka's consumption (offset) as a record to determine which data is processed to, but write it later In the process of the storage side, it is impossible to ensure that the data written to each storage side is consistent; the other is an error retry (retry) mechanism, after an error occurs in the data writing to t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & AuthorityApplications(China)
IPC IPC(8): G06F11/07G06F11/30G06F16/23
CPCG06F11/0787G06F11/3034
Inventor火一莽高魁万月亮
OwnerRUN TECH CO LTD BEIJING