Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

A data processing method and device

A data processing and data technology, applied in the field of network applications, can solve problems such as taking up a lot of time and reducing data storage speed, and achieve the effect of saving time, improving processing speed, and improving storage speed.

Active Publication Date: 2017-04-12
BEIJING QIHOO TECH CO LTD
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] When there is a lot of input data, it will take a lot of time to id each data (such as the key in each key-value pair), which will reduce the data storage speed

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A data processing method and device
  • A data processing method and device
  • A data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0116] In this embodiment, the original data (vectors) in Table 1 on the left are continuously idized and converted into the form in Table 2 on the right.

[0117]

[0118] In this example, 2 nodes (Node0, Node1) are used, and each node has 2 threads (Thread0, Thread1). According to the number of nodes, the original data can be divided as shown in Table 3:

[0119] Table three

[0120]

[0121] Referring to Table 3, we can see that each node processes 5 pieces of data. by the formula It can be calculated that the starting number of Node0 is 0, and the starting number of Node1 is 5.

[0122] Each Thread sequentially accesses the data to be processed by the Thread, and uses the atomic operation to increment the start number each time one is processed. For example, Thread0 of Node0 processes the first data (10,3), then the number is (0,3), and the number is incremented by atomic_inc to 1. Thread1 of Node0 processes the first data (40,6), and the number is (1,6).

[01...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a method and a device for processing data. The device comprises a grouping module, a processing module and a memory module, wherein the grouping module is configured to group the input data, wherein the data group numbers are determined according to nodes which process the data; the processing module is configured to distribute a group of data to be processed to each node sequentially and utilizes the nodes to carry out parallel continuous identification (id) processing on the respective corresponding data, wherein the continuous id processing is that the keyword in the data is replaced by integers of 0-N; the memory module is configured to store the data after the continuous id processing, wherein the processing module is also configured to calculate the initial id number of arbitrary node; and the node data is identified according to the initial id number of the node. After the device is adopted, the data can be subjected to parallel continuous id processing, so that the storage time is saved.

Description

technical field [0001] The invention relates to the field of network applications, in particular to a data processing method and a data processing device. Background technique [0002] In massively parallel computing, the input data is usually a series of key-value (key-value) pairs. After a series of calculations, the new value corresponding to each key is output. In order to facilitate addressing and save storage space, the input data can be continuously idized, that is, the key in each key-value pair is replaced by a 0-N integer. [0003] When there is a lot of input data, iding each data (such as the key in each key-value pair) will take a lot of time, which will reduce the data storage speed. Contents of the invention [0004] In view of the above problems, the present invention is proposed to provide a data processing device and a corresponding data processing method that overcome the above problems or at least partially solve the above problems. [0005] Accordin...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
Inventor 齐路何锐邦唐会军
Owner BEIJING QIHOO TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com