Unlock instant, AI-driven research and patent intelligence for your innovation.

A data transfer method, device, electronic equipment and storage medium

A technology of data and equipment, applied in the field of data transfer methods, devices, electronic equipment and storage media, can solve the problems of low data transfer efficiency, poor scalability, high risk, etc., and achieve the goal of improving efficiency, reducing risks and reducing costs Effect

Active Publication Date: 2021-10-29
TENCENT TECH (SHENZHEN) CO LTD
View PDF9 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] Embodiments of the present invention provide a data transfer method, device, electronic equipment, and storage medium to solve the problems of low data transfer efficiency, high risk, and poor scalability in the prior art

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data transfer method, device, electronic equipment and storage medium
  • A data transfer method, device, electronic equipment and storage medium
  • A data transfer method, device, electronic equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0055] In order to facilitate the understanding of the embodiments of the present invention, several concepts are briefly introduced below:

[0056] Message Queue (Message Queue, mq): A circular message queue implemented using shared memory (shm).

[0057] chubby: Distributed lock service, providing a file system-like interface and a coarse-grained lock mechanism.

[0058] Hadoop: It is a software platform for developing and operating large-sc...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The present invention relates to the technical field of the Internet, in particular to a data transfer method, device, electronic equipment and storage medium, which determines the service name of the data source of the data to be read, and obtains the current record corresponding to the service name from a preset database index value; according to the index value, read data from the first storage system corresponding to the service name; when the preset condition is met, write the read data into the second storage system; wherein, the first storage system and The second storage system belongs to different storage systems. In this way, the index value corresponding to each service name can be uniformly recorded, and according to the index value currently recorded, the data is read from the corresponding first storage system and directly written into the second storage system. In the storage system, it does not need to be stored in the local disk first, reducing the risk of writing to the local disk, and can be applied to the first storage system of any service name. It does not need to be developed separately. It has strong scalability, improves efficiency, and reduces cost.

Description

technical field [0001] The present invention relates to the technical field of the Internet, in particular to a data transfer method, device, electronic equipment and storage medium. Background technique [0002] Currently, in the Internet search business, multiple third-party services will push their own web page data to the search service provider for inclusion. The search service provider usually caches or permanently stores the received data in the form of a message queue (Message Queue, mq) for use by technical personnel. When technicians process data, such as data sorting, they use hadoop to perform calculations, but Hadoop does not support mq, so they need to read data from mq and upload the read data to hdfs before they can be read from hdfs Pull data for processing. [0003] In the prior art, the method of transferring data in mq to hdfs is usually written by technicians independently, reads data from mq and stores them in the local disk, and then uploads them to ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/13G06F16/16G06F16/182G06F9/54G06F9/52
CPCG06F9/524G06F9/546G06F2209/548
Inventor 孙凯李辉邵杰
Owner TENCENT TECH (SHENZHEN) CO LTD