Data merging method, device, electronic device and storage medium

A data and data acquisition technology, applied in the field of big data, can solve the problems of large memory overhead, inability to merge, poor real-time data transmission, etc., and achieve the effect of meeting merge requirements, long merge cycle, and solving large-scale data disorder

Active Publication Date: 2021-11-16
BEIJING QIANXIN TECH
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

There are the following technical problems. The out-of-order data cannot be merged, and during the merge cycle, the data is accumulated in the memory, the memory overhead is large, and the data needs to wait until the end of the window to output the data, and the real-time performance of data transmission is poor.

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data merging method, device, electronic device and storage medium
  • Data merging method, device, electronic device and storage medium
  • Data merging method, device, electronic device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0034] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

[0035] figure 1 A flow chart of the data merging method provided by the embodiment of the present invention is shown. like figure 1 As shown, the data merging method provided by the embodiment of the present invention includes the following steps:

[0036] Step 101: Acquiring data and processing a rule set of data, wherein the rule set inc...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

Embodiments of the present invention provide a data merging method, device, electronic equipment, and storage medium. Wherein, the data merging method includes: acquiring data and processing a rule set of data, wherein the rule set includes matching rules; performing rule matching on the data, if the data hits any match in the rule set rule, adding a corresponding rule identifier of the matching rule to the data, and generating a merge identifier for the data based on the data time of the data, the preset merge cycle and the rule identifier, if the data does not match the If any matching rule in the rule set is used, the data is output to an external storage system; and the data is merged based on the merge identifier. The embodiment of the present invention can solve the problem of data merging when data in a large range is out of sequence, and has the advantages of high real-time performance of data transmission and low memory overhead.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data merging method, device, electronic equipment and storage medium. Background technique [0002] At present, for data merging, the technical means usually adopted is to use the window mechanism to realize, for example: when a piece of data arrives at the window operator, the window operator will establish a window according to the time of the current data and the merging period, and then follow up according to Matching rules accumulate data into this window for merging, and then, when the window ends, process the data in the window and output it to the external storage system. There are the following technical problems. The out-of-order data cannot be merged, and during the merge cycle, the data is accumulated in the memory, the memory overhead is large, and the data needs to wait until the end of the window to output the data. The real-time performance of data tran...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F3/06
CPCG06F3/0608G06F3/061G06F3/0638G06F3/067
Inventor 常月
Owner BEIJING QIANXIN TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap