Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data consistency detection method, device and equipment and computer storage medium

A detection method and consistency technology, applied in the direction of structured data retrieval, digital data information retrieval, calculation, etc., can solve the mismatch of the resource data volume of the consistency detection system, increase the complexity of the consistency detection business logic, and the lack of data De-fragmentation and other problems to achieve the effect of improving scalability, solving data fragmentation, and reducing data volume

Pending Publication Date: 2019-07-16
中国移动通信集团重庆有限公司 +1
View PDF7 Cites 3 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

The common problem faced by such solutions is that they cannot effectively cope with the huge increase in data, and often can only reduce the amount of data to be processed in a single batch by subdividing the business scenarios generated by the data, and the expansion performance is poor; Secondly, in the face of the actual situation of data fragmentation caused by high-frequency transactions, data fragmentation cannot be effectively achieved, and the system resources consumed by consistency detection do not match the amount of data that needs to be processed.
Although the application of cloud computing and other technologies can ensure sufficient system resources to meet the requirements of consistency detection for system resources, it will also greatly increase the business logic complexity of consistency detection

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data consistency detection method, device and equipment and computer storage medium
  • Data consistency detection method, device and equipment and computer storage medium
  • Data consistency detection method, device and equipment and computer storage medium

Examples

Experimental program
Comparison scheme
Effect test

specific example 1

[0112] Specific example 1, the detection process when the first set of data is generated:

[0113] Step 1, the data acquisition and analysis module completes the ETL processing of data according to the standard;

[0114] Step 2, the data grouping module groups the received data according to transaction ID, time period, business category, data source category and other dimensions, and then splits the original data group into integer value groups and non-integer value groups according to the business data rounding rules ;

[0115] Step 3, the index control module assigns a unique index ID consisting of public segments and private segments to each set of data;

[0116]Step 4, because the first group of data will not be queried from the release control module to the historical associated data group, so it is only necessary to complete the fragmentation compression of the data in the group and the carry compression of the non-integer value group;

[0117] Step 5, after the publis...

specific example 2

[0119] Specific example 2, the detection process when there is fragment compression between groups:

[0120] Step 1, the data group processed by the collection, analysis, grouping, index control and other modules is normally submitted to the fragment compression module;

[0121] Step 2, the fragment compression module completes the fragment compression of the data in the group;

[0122] Step 3, the fragment compression module initiates a query request for the associated data group to the release control module;

[0123] Step 4, the release control module matches the index of the associated data group that is inconsistent and the transaction is not completed according to the index public segment information, and returns it to the fragment compression module;

[0124] Step 5, the fragment compression module completes the data fragment compression between groups, and submits the compressed data group to the release control module;

[0125] Step 6, release the control module gen...

specific example 3

[0128] Specific example 3, the life cycle control process of the unfinished data group:

[0129] Step 1, the subscription control module queries the consistency detection task subscription list 1 in real time or quasi-real time;

[0130] Step 2. For detection tasks that are in the waiting state and have exceeded the maximum time, generate a subscription relationship cleaning task list; for detection tasks that are silent and have exceeded the maximum silent time, reset their status to waiting, and set the maximum long waiting time;

[0131] In step 3, the subscription control module completes the cleaning of the subscription relationship.

[0132] The data consistency detection device and method provided by the embodiments of the present invention make up for the shortcomings of the traditional consistency detection scheme, can effectively reduce the amount of data to be detected, and improve the scalability of the detection system at the same time.

[0133] Figure 9 A sch...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data consistency detection method, device and equipment and a computer storage medium. The detection method comprises the following steps: receiving to-be-detected data, anddividing the to-be-detected data into a plurality of data groups; allocating indexes to the data groups; according to the indexes, retrieving a data set needing fragment compression, and performing fragment compression on the data in the set; and converting the to-be-detected data after fragment compression into a detection task according to the dynamically generated subscription list, and publishing the detection task to a corresponding detection node according to a subscription relationship to carry out consistency detection. According to the embodiment of the invention, by grouping the dataand establishing the index, the detection processing on each piece of data is avoided, so that the data volume needing to be detected is reduced; the problem of data fragmentation is solved by performing fragmentation compression on the data in the group; through a subscription / release mechanism, full-process recording and tracking of transaction consistency detection are realized, and meanwhile,the expandability of the detection system is improved.

Description

technical field [0001] The invention belongs to the field of data processing systems, and in particular relates to a data consistency detection method, a data consistency detection device, computer equipment and a computer storage medium. Background technique [0002] All transactions generate data. The role of data is firstly as the core component of the transaction, which directly determines the success or failure of the transaction, and secondly, as a part of the transaction log, it is used for post-event analysis and inspection of transaction quality. For data generated by the same transaction, with similar characteristics but different objects and transmission paths, we usually call it same-source but different-path data. [0003] When the data is used as part of the transaction log for the inspection of transaction quality, the purpose is to detect data anomalies in time and avoid data exceeding the limit of the rules. The core of transaction quality inspection is th...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/23G06F16/174
CPCG06F16/1744G06F16/2365
Inventor 陈海龙
Owner 中国移动通信集团重庆有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com