Big data acquisition method based on large model technology

Abstract

The invention discloses an adaptive big data acquisition method driven by a big language model, which is oriented to heterogeneous and changeable data sources and realizes high-quality, low-cost and compliant acquisition through a'planning-execution-evaluation-optimization 'closed loop. According to the method, LLM is used for task analysis, strategy generation and structured extraction, and lake entry and traceable treatment are completed in combination with quality evaluation, semantic deduplication and privacy desensitization. According to the technical scheme, the method comprises the steps of 1, task analysis and strategy planning; and 2, multi-source parallel acquisition and incremental updating. And 3, performing mixed extraction and mode alignment, and performing quality evaluation and semantic deduplication. And 4, compliance desensitization, auditing lake entering and feedback optimization.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data engineering and artificial intelligence, and relates to big data collection, governance and compliance, high-quality and low-cost collection and lake entry scenarios for heterogeneous data sources. For the use of large models, whether local models or third-party APIs, they are used in the form of API. BACKGROUND

[0002] Existing collection relies on fixed rules and scripts, which cannot adapt to frequent changes in data sources and the increasing proportion of unstructured data. The maintenance cost is high, the quality is unstable, and there is a lack of compliance audit, making it difficult to achieve a dynamic balance between cost, quality and timeliness. SUMMARY

[0003] The application takes the "planning-execution-evaluation-optimization" closed loop driven by LLM as the core, realizes the integrated process of task analysis, strategy generation, multi-source parallel collection, mixed extraction, quality evaluation, semantic deduplication, privacy desensitization and lake entry audit, and continuously optimizes the strategy and cost online.

[0004] Extended application scenarios: policy and information extraction, product and price collection, log and Internet of Things data aggregation, industry data compliance lake entry, knowledge graph update and retrieval enhancement, and multi-source heterogeneous scenarios.

[0005] Core technology: Few-shot prompt and tool call task analysis; adaptive concurrency and throttling; incremental water level management; LLM+rule mixed extraction and Schema alignment; semantic deduplication; privacy identification and desensitization; full-link audit and feedback optimization. TECHNICAL SCHEME

[0006] A big data collection method based on large model technology, characterized by the following steps: Requirement analysis and strategy planning: LLM generates a plan containing source, field, mode, strategy and compliance constraints; Multi-source parallel collection: dynamic concurrency, retry and proxy switching, supporting incremental update; Mixed extraction and alignment: block extraction, rule verification and JSON Schema alignment; Quality governance and lake entry: quality scoring, semantic deduplication, compliance desensitization and audit lake entry.

[0007] A big data collection method based on large model technology, characterized by including: Dispatching features: priority queue, throttling and fallback strategy; Extraction features: multi-modal support and cross-section consistency verification; Governance features: fingerprint + vector deduplication, conflict merging and versioned traceability.

[0008] BRIEF DESCRIPTION OF DRAWINGS Figure 1 The flow chart of the method of the present application is shown, which shows the whole process from task analysis, strategy planning, parallel collection, extraction and management to lake entry and feedback optimization. DETAILED DESCRIPTION

[0009] Analyze the "new energy policy" task, plan the mixed collection of web pages / API and throttling strategy; Execute parallel collection and schedule by domain name and priority, and automatically fallback to proxy when speed limit is encountered; Text is divided by section, LLM extraction (title, main body, terms, time limit, etc.) and rule verification; De-duplication is performed using SimHash and vector similarity, and time and credibility are weighted to solve conflicts; PII is identified and masked, and audit is generated for source URL, timestamp, prompt word and model version; Data is entered into the lake and indexed, and samples-strategy-quality are returned simultaneously for online optimization.

[0010] Beneficial effects: under the premise of ensuring compliance and traceability, adaptive optimization of collection strategy and high-quality structured output are realized, maintenance cost and API cost are significantly reduced, and coverage, completeness and consistency are improved.

Claims

1. A large model technology-based big data collection method, characterized by Comprise: A large language model analyzes natural language tasks to generate an initial plan, including a data source set, a field set, an extraction pattern, a collection strategy, and compliance constraints. Based on the strategy, multi-source parallel adaptive collection is performed, and the concurrency, retry, proxy, rendering, and incremental water level are dynamically adjusted. Block and mixed extraction are performed on the original data and aligned to the target pattern, the quality score is calculated, and fingerprint and semantic similarity deduplication and conflict merging are performed. Privacy identification and desensitization are performed, audit logs are generated, and the governed data is written to the lake warehouse, index, and knowledge graph. At the same time, data-strategy-quality information is used for online optimization.

Images