Data processing method and device

A data processing and processor technology, applied in the field of distributed computing, can solve problems such as slow computing speed, inability to use, versatility, lack of compatibility support, etc., and achieve the effect of increasing computing speed, improving versatility, and speeding up computing speed

Active Publication Date: 2016-11-23
BEIJING QIHOO TECH CO LTD
View PDF3 Cites 16 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Using a platform that uses CPU computing nodes at the bottom layer may cause the problem of slow computing speed due to CPU performance limitations
[0004] In addition, some professional fields have recently launched some dedicated platforms, but the computing nodes used are also a single model. In CPU (Central Processing Unit, central processing unit), GPU (Graphic Processing Unit, graphics processing unit), FPGA (Field-Programmable Gate Array, field programmable gate array), DSP (digi

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device
  • Data processing method and device
  • Data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Example Embodiment

[0072] Example one

[0073] Reference figure 1 , Which shows a schematic flow chart of a data processing method of the present invention. It can include:

[0074] Step 110: According to the input interface of the vector template corresponding to the data type of the input data, load the input data into the vector template to obtain a calculated vector;

[0075] In the embodiment of the present invention, various types of vector templates Vector are set and constructed in advance according to different types of data. Then users can select the corresponding vector template to input data according to their needs. The Vector includes: template vectors including FixValueVector, SparseValueVector, SparseTextVector, VariableLenVector, Var, PVar, SparseMatrix, etc. Among them, SparseValueVector represents a vector with sparse data, which is used for input data with sparse data; FixValueVector represents a vector with dense data, which can be used for data dense input data; SparseTextVecto...

Example Embodiment

[0141] Example two

[0142] Reference figure 2 , Which shows a schematic flow chart of a data processing method of the present invention. It can include:

[0143] Step 210: According to the input interface of the vector template corresponding to the data type of the input data, load the input data into the vector template to obtain a calculated vector;

[0144] Step 212: Split the corresponding calculation vector in the processing logic into each calculation sub-vector.

[0145] Combine Figure 2A , To describe the embodiments of the present invention, Figure 2A It is the computational logic architecture model of the present invention.

[0146] In the embodiment of the present invention, the user first edits the entire calculated user logic on the application code logic side, and compiles the input data. The user logic includes each OP operator and execution order, such as OP operator A-OP operator B-OP operator C. Then the computing platform of the embodiment of the present invent...

Example Embodiment

[0180] Example three

[0181] Reference image 3 , Which shows a schematic structural diagram of a data processing device of the present invention. It can include:

[0182] The template vector processing module 310 is adapted to load the input data into the vector template to obtain the calculated vector according to the input interface of the vector template corresponding to the data type of the input data;

[0183] The calculation vector allocation module 320 is adapted to split the corresponding calculation vector in the processing logic into each calculation sub-vector;

[0184] The OP operator splitting module 330 is adapted to obtain the OP operator currently used to calculate each calculation sub-vector, and determine whether the OP operator can be split into a combination of each sub-operator according to a preset sub-operator library ;

[0185] The sub-operator calculation and selection module 340 is adapted to split the OP operator into a combination of each sub-operator if ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method and device and relates to the technical field of distributed computation. Computational node mixed establishment distributed computation systems of multiple processor types can be adopted, for example, computational nodes and the like CPU processors are adopted in the whole computation system in a mixed mode, preset corresponding types of vector templates are adopted to load input data of a user according to the data types, then before computation vectors are split into computation sub-vectors to be distributed to the computational nodes for computation, OP operators edited by the user are firstly split into combinations of sub-operators, the computational nodes matched with the processor types during computation of the computational nodes are judged, and then data is sent to corresponding computational nodes for computation. A computation platform is achieved in a heterogeneous mode, the computational nodes of different processor types are used in parallel, and the universality and compatibility are improved.

Description

technical field [0001] The invention relates to the technical field of distributed computing, in particular to a data processing method and device. Background technique [0002] As big data analysis research has become a hot topic, MPI-based distributed memory computing platforms have once again attracted attention in the industry in recent years. [0003] For the analysis and processing of massive big data, the popular distributed computing platforms generally include the following types: MapReduce computing in hadoop, spark, streaming computing (represented by storm), and memory computing based on mpi. But the bottom layer of the hardware of most of these platforms adopts CPU as the core computing node. Using a platform that uses CPU computing nodes at the bottom layer may cause the problem of slow computing speed due to CPU performance limitations. [0004] In addition, some professional fields have recently launched some dedicated platforms, but the computing nodes use...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/38
Inventor 白明
Owner BEIJING QIHOO TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap