Unlock instant, AI-driven research and patent intelligence for your innovation.

Data file processing method and system

A technology of data files and processing methods, which is applied in the field of communication, can solve problems such as long loading time, memory usage, and low loading efficiency, and achieve the effect of improving loading efficiency and reducing memory usage

Active Publication Date: 2017-05-24
TENCENT TECH (SHENZHEN) CO LTD
View PDF5 Cites 8 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] During the research and practice of the prior art, the inventor of the present invention found that the memory data in the prior art can only be used in a single process, and if multiple processes need to use the same data, they need to be loaded and updated separately. This will lead to additional memory usage; and the data is a plaintext data structure, which takes a long time to load, resulting in low loading efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data file processing method and system
  • Data file processing method and system
  • Data file processing method and system

Examples

Experimental program
Comparison scheme
Effect test

no. 1 example

[0033] In this embodiment, description will be made from the perspective of data file processing, and the data file processing system may specifically be integrated in devices such as servers.

[0034] A method for processing a data file, comprising: obtaining a data file on a disk, wherein the data file is stored in a data structure of a shared memory; using memory mapping to load the data file into the shared memory, and performing a process on the shared memory Initialization; record data update information based on the loading process, the data update information includes the file name of the data file and first time information, the first time information is the time when the loading process is loaded; perform data file processing according to the data update information Read processing.

[0035] see Figure 1b , Figure 1b It is a schematic flowchart of the data file processing method provided by the first embodiment of the present invention. The method includes:

[...

no. 2 example

[0054] According to the method described in the first embodiment, an example will be given below for further detailed description.

[0055] The data file processing system includes: a data management module, a data usage module and an update detection module; first, the data management module has mapped the data files on the disk to the shared memory, and recorded the data update information of the process. Secondly, the data usage module also maps the data file to the shared memory according to the data update information; after the data loading is completed, the update detection module can further update and detect the disk file in real time, so that the data management module and the data reading module can perform data processing based on the data. Update and load.

[0056] The data involved in the update loading of the system includes: data files on the disk, shared memory data and recorded data update information, and a cyclic redundancy check file (CRC, Cyclic Redundanc...

no. 3 example

[0094] In order to better implement the data file processing method provided by the embodiment of the present invention, the embodiment of the present invention further provides a system based on the above data file processing method. The meanings of the nouns are the same as those in the above data file processing method, and for specific implementation details, please refer to the description in the method embodiments.

[0095] see Figure 4 , Figure 4 The schematic structural diagram of the data file processing system provided by the embodiment of the present invention may specifically include a data management module 401 and a data reading module 402 .

[0096] Wherein, the data management module 401 is used to map the data files on the disk into the shared memory, and initialize the shared memory; record data update information based on the loading process, so as to realize the loading process of the data files. It can be as follows:

[0097] The data management modul...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data file processing method and system. The method comprises the steps of obtaining a data file in a disk, wherein the data file is stored by a data structure of a shared memory; loading the data file to the shared memory in a memory mapping manner, and initializing the shared memory; recording data updating information based on the loading process, wherein the data updating information comprises a file name of the data file and first time information, and the first time information is loading time of the loading process; and performing reading processing on the data file according to the data updating information. According to the method and the system provided by embodiments of the invention, the data file, adopting a plaintext data structure, in the disk is stored by the data structure of the shard memory and loaded to the shared memory, so that the loading efficiency is improved; and a one-loading and multi-use function is supported, namely, a plurality of processes use the same shared memory data through the shared memory, so that the additional memory occupation is greatly reduced.

Description

technical field [0001] The invention belongs to the technical field of communication, and in particular relates to a data file processing method and system. Background technique [0002] With the rapid development of Internet technology, a large amount of data is required to support decision-making in a large number of services such as search string analysis and error correction in search business and recommendation service in personalized recommendation business. In the program, online services need to load a large amount of data, and each processing requires a large number of table lookup operations. At the same time, data needs to be updated at a certain frequency to adapt to changes. The update frequency can be real-time (second level), quasi real-time (minute level) or regular (day level) update. [0003] In the existing technology, each service process is usually responsible for loading and updating data by itself. Generally, the process reads the data file and build...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 王刚
Owner TENCENT TECH (SHENZHEN) CO LTD