Big data text quick processing method

A processing method and big data technology, applied in the direction of electrical digital data processing, special data processing applications, natural language data processing, etc., can solve problems affecting data processing speed, different processing processes, long time, etc., to improve data processing speed , Improve processing speed and improve efficiency

Inactive Publication Date: 2018-05-01
BEIJING AEROSPACE AIWEI ELECTRONICS TECH LIMITED +1
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The defects or deficiencies of this traditional method are as follows: firstly, when the text content is too large, since the traditional processing method is to process by row, it will take a long time and seriously affect the data processing speed; secondly, due to the different processing results, the processing process different, which also affects the processing speed of the data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data text quick processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0018] In order to make the purpose, content, and advantages of the present invention clearer, the specific implementation manners of the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

[0019] Such as figure 1 As shown, the embodiment of the present invention provides a kind of fast processing method of big data text, comprises the following steps:

[0020] Step 1. Read the data file: read the data content in the file through the specified file name;

[0021] Step 2, data slicing: process the file data read in step 1 into blocks, and mark the data blocks after block processing;

[0022] Step 3, thread pool processing: put the data block into the thread pool and perform data processing according to preset requirements;

[0023] Step 4. Process and save data by block: save the processed data block locally according to the mark of the current data block;

[0024] Step 5, traverse the data block: detec...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a big data text quick processing method, and belongs to the technical field of computer information processing. The method can effectively improve the use rate of a CPU through a thread pool, can improve the processing speed of big text data, can mark data blocks on which partitioning is performed in the thread pool so as to effectively prevent data confusion, can performlocal saving on the processed data blocks in the thread pool according markers, can monitor the use state of the thread pool, can read the traversed saved and marked data blocks when the thread pool is idle, and can merge and save the data blocks; and in this way, the space can be effectively utilized to shorten the processing time, and then the efficiency is improved. The method can improve the data processing speed through a multithread mode by means of the thread pool, can use the space to save the time, can improve the data merge speed by using the high efficiency read-write speed of a stream, and can greatly improve the processing speed of a big data text through combination of the thread pool technology and the space storage technology.

Description

technical field [0001] The invention relates to the technical field of computer information processing, in particular to a method for quickly processing large data texts. Background technique [0002] For large data text, the traditional processing method is to read the text, and then process the data text into the expected effect and then write it to the file. The defects or deficiencies of this traditional method are as follows: firstly, when the text content is too large, since the traditional processing method is to process by row, it will take a long time and seriously affect the data processing speed; secondly, due to the different processing results, the processing process Different, this will also affect the processing speed of the data. Contents of the invention [0003] (1) Technical problems to be solved [0004] The technical problem to be solved by the present invention is: how to improve the processing speed of large data texts. [0005] (2) Technical solu...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/21
CPCG06F40/10G06F40/117
Inventor 刘晓冬张琍蔡娜刘磊王旭初王祎鹏
Owner BEIJING AEROSPACE AIWEI ELECTRONICS TECH LIMITED
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap