Supercharge Your Innovation With Domain-Expert AI Agents!

Data processing method and device

A data processing and data volume technology, applied in the field of data processing, can solve the problems of increasing data volume, reducing data collection efficiency, increasing storage space, etc.

Inactive Publication Date: 2018-03-09
NEW H3C BIG DATA TECH CO LTD
View PDF6 Cites 5 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] But generally in the process of data collection, the user's data is not allowed to add irrelevant fields. In addition, in the above scheme, adding the id column will lead to an increase in the amount of data, resulting in an increase in storage space. Reduce data collection efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device
  • Data processing method and device
  • Data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0052] figure 1 A flowchart showing a data processing method according to an embodiment of the present invention. Such as figure 1 As shown, the data processing method can be applied to data processing equipment. The data processing device may specifically be a terminal device equipped with a data collection and processing platform, for example, a desktop computer, a personal computer, and the like. The data processing method in the embodiment of the present invention includes the following steps:

[0053]Step 101. Obtain a data column and a first number N of fragments for dividing the data column; wherein, the N is an integer greater than 0.

[0054] For example, a data column can be a feature that can represent a type of data, for example: data is data stored in a data table for data analysis, and a data column can be a field in a data table. Exemplarily, in a certain province When the population age distribution data is collected, the data column can be age; or, when th...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

An embodiment of the invention relates to a data processing method and device. The method comprises steps as follows: data columns and first sharding number N of division of the data columns are acquired, wherein N is an integer larger than 0; the data columns are subjected to sharding processing according to the first sharding number N, and N first data fragments are obtained; whether each of theN first data fragments meets the preset sharding rule is judged, the first data fragments meeting the sharding rule are subjected to sharding processing again, and second data fragments are obtained;when the second data fragments do not meet the sharding rule, the first data fragments not meeting the sharding rule and the second data fragments are subjected to data processing. According to the data processing method and device in the embodiment of the invention, the problem of local hot spots can be solved, and data collecting efficiency is improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data processing method and device. Background technique [0002] With the development of network technology (Internet of Things, cloud computing, cloud storage, etc.), it is accompanied by the generation of massive data and the problem of how to process the data. [0003] Due to the large amount of data, in the process of data collection, distributed collection is often used for data collection to improve the efficiency of data collection. However, the distributed collection method has local hotspot problems, such as large amount of data in some tasks and small amount of data in some tasks caused by uneven data distribution, which leads to low resource utilization and reduced data collection efficiency. [0004] In order to make the data distribution even, before or during the data collection, an incremental id column can be added to the source table of the dat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F19/00
CPCG16Z99/00
Inventor 楼浩盛
Owner NEW H3C BIG DATA TECH CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com