Multi-source data integration and structured storage method for urban international propagation influence

Abstract

The invention discloses an urban international propagation influence multi-source data integration and structured storage method, which is realized based on a five-layer decoupling architecture originally embedded on a large language model. The method comprises the following steps: S1, collecting multi-source heterogeneous data such as social media and news, retaining original data, collecting metadata, and carrying out partitioned storage; s2, cleaning and annotating the data through a large language model, and generating a semantic-rich record containing semantic annotations and quality confidence fields; s3, constructing a topic model containing details and pre-aggregated data, and adding a data quality watermark; s4, dynamically generating a materialized view based on the query hot spot, and executing cross-source intelligent bit complementing when the data source is missing; and S5, packaging data product packages with different granularities and providing a data service interface. According to the method, efficient conversion of data from an original state to semantic enhanced assets is realized, and the method has the advantages of low cost, high precision and high adaptability.

Description

[0001] This invention relates to the fields of big data processing and artificial intelligence technology, specifically to a method for integrating and storing multi-source data on a city's international communication influence. Background Technology

[0002] The assessment of a city's international communication influence is a typical scenario for multi-source heterogeneous data analysis. It requires the integration of data from multiple channels such as social media, news portals, search engines, and tourism review platforms. These data have mixed formats, inconsistent field definitions, and contain content in multiple languages ​​such as English, Arabic, and Russian, which poses a huge challenge to data processing.

[0003] Existing technologies suffer from several structural defects: First, data sources are fragmented and semantic noise is prominent. Slang and cultural metaphors in cross-language texts are difficult to clean and identify using traditional rules, resulting in a false deletion rate exceeding 30%. Low-resource language processing is even more challenging, and sentiment analysis accuracy is insufficient. Traditional keyword matching or shallow machine learning models have low cross-language accuracy and are ineffective at recognizing implicit and mixed sentiments. While general-purpose LLM (Large Language Model) is used, it is costly, has high latency, and is not deeply coupled with the ETL (Extract, Transform, Load) process. Second, the ETL process is rigid. Fixed cleaning rules cannot adapt to changes in data noise under sudden events. Intelligent supplementation relies solely on simple statistical interpolation and lacks semantic-level correlation completion capabilities. Furthermore, the degree of data assetization is low. The intermediate layer data in the traditional three-layer architecture lacks semantic annotation and quality marking. Intermediate process data is untraceable, making it difficult to achieve gradient asset accumulation and resulting in low value density. Third, the inter-layer coupling in the architecture is high. Schema mapping relies on hard coding, resulting in poor scalability and a lack of unified quality monitoring and lineage tracing mechanisms, making it difficult to locate data quality problems.

[0004] These issues mean that existing solutions cannot meet the requirements of accurate data processing, dynamic adaptability, and asset utilization for assessing a city's international communication influence. There is an urgent need for a new ETL process solution that integrates advanced AI technology to address these problems. Summary of the Invention

[0005] The purpose of this invention is to provide a method for integrating and storing structured data on a city's international communication influence from multiple sources. This invention can effectively transform and efficiently utilize data from its raw state into semantically enhanced assets, offering advantages such as low cost, high precision, and high adaptability.

[0006] The technical solution of this invention: A method for integrating and storing multi-source data on the international communication influence of cities. This method is based on a five-layer decoupling architecture with native embedding of a large language model. The five-layer decoupling architecture, from bottom to top, consists of a source-attached buffer layer, a semantic enhancement operation layer, a topic model layer, an intelligent marketplace layer, and a service assetization layer. The method includes the following steps: S1: Collect multi-source heterogeneous data and store it in the source buffer layer. The multi-source heterogeneous data includes social media data, news data, search index data, and travel review data. When storing, the original data and its collected metadata are retained and organized using a partitioned directory. S2: Push the data from the source buffer layer to the semantic enhancement operation layer, use a large language model to clean and semantically annotate the data, and generate a semantically rich record containing semantic annotation fields and quality confidence fields; S3: Based on the semantically enhanced operation layer outputting the semantically rich records, construct a topic model containing detailed data and pre-aggregated summary data in the topic model layer, and add data quality watermarks to the data partitions; S4: Based on query hotspots, dynamically generate materialized views in the smart marketplace layer and perform cross-source intelligent fill-in when the data source is missing; S5: At the service assetization layer, data from each level is encapsulated into data product packages of different granularities, and corresponding data service interfaces are provided. In the above-mentioned method for integrating and storing multi-source data on the international communication influence of cities, in step S1, the source buffer layer uses a dual schema storage of raw data and collected metadata. The collected metadata includes the collection timestamp, API version, and data freshness TTL. The raw data is stored in three-level partitions according to the data source type, collection batch, and time. The partition directory adopts a three-level structure of data source type, collection batch, and collection time, with the partition granularity accurate to the hour level.

[0007] The aforementioned method for integrating and structuring multi-source data on the international communication influence of cities, specifically includes the following step S2: Cleaning and semantic annotation using a large language model. Use a large language model that has been domain-adaptively fine-tuned as the base model; A hybrid cueing engineering strategy is adopted, which selectively applies hard cues, few-sample cues, or thought chain cues based on the complexity and value of the text. A semantic caching library is built. A semantic fingerprint is calculated for new text. If the Hamming distance between the new text and the semantic fingerprint recorded in the cache is less than a set threshold, the cached result is directly reused.

[0008] The aforementioned method for integrating and structured storing multi-source data on the international communication influence of cities, and the hybrid prompting engineering strategy include: For regular data, hard hints are provided using a fixed template without contextual examples. For complex text, a few-sample hint containing examples from several domains is used for processing; For high-value texts, a thought chain prompting method that requires step-by-step reasoning using a large language model is employed.

[0009] The aforementioned method for integrating and storing multi-source data on the international communication influence of cities, wherein the semantically rich record generated in step S2 contains a four-tuple structure: original field, standard field, semantic annotation field, and quality confidence field; the semantic annotation field includes at least machine-translated text, sentiment score, entity recognition result, and noise label; and a 128-bit semantic fingerprint is generated for each record for deduplication and association.

[0010] The aforementioned method for integrating and structuring multi-source data on the international communication influence of cities further includes a dynamic semantic filtering step in step S2: Calculate the semantic entropy value for each batch of data; Monitor the offset of the semantic entropy value relative to the historical baseline; When the offset exceeds the preset threshold, the noise judgment threshold in the data cleaning process is automatically adjusted.

[0011] In the aforementioned method for integrating and structuring multi-source data on the international communication influence of cities, step S3 includes a topic model layer comprising a detailed data layer and a summary data layer. The detailed data layer is partitioned by time, city, and topic. The summary data layer stores pre-calculated aggregated indicator cubes.

[0012] The aforementioned method for integrating and structured storing multi-source data on the international communication influence of cities, wherein the intelligent supplementation mentioned in step S4 includes at least one of the following mechanisms: Homologous historical fill: Predicting missing values ​​based on time series data from historical sources; Cross-source semantic padding: Filling in records with semantic similarity by retrieving semantic fingerprints from other data sources; Knowledge graph supplementation: Generate synthetic labeled data based on the historical sentiment baseline of the city entity knowledge graph and current hot events.

[0013] In the aforementioned method for integrating and storing multi-source data on the international communication influence of cities, step S5 involves an encapsulated data product package accompanied by a data asset certificate. The data asset certificate includes data lineage information, quality score details, processing version information, and usage license information.

[0014] The aforementioned method for integrating and storing multi-source data on the international communication influence of cities involves the five-layer decoupled architecture, where the layers are linked through an event-driven mechanism. Data flow between layers is achieved by transmitting quality scoring information through metadata services, thus forming a quality influence chain.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention efficiently solves the challenges of multilingual semantic noise recognition and cross-lingual sentiment analysis by employing a domain-adaptive fine-tuning large language model and a hybrid prompting engineering strategy. It accurately captures complex semantic information such as implicit and mixed sentiment, significantly reducing the false deletion rate of traditional rule-based cleaning. This invention is compatible with diverse data sources such as social media, news, search indices, and travel reviews, supports multilingual and long text processing, and flexibly supports the calculation of multi-dimensional indicators such as online dissemination, media reports, and international visitors. It provides comprehensive and accurate data support for assessing a city's international communication influence and adapts to the personalized assessment needs of different cities.

[0016] 2. The dynamic threshold adjustment algorithm based on semantic entropy monitoring can automatically adapt to data distribution drift scenarios, breaking away from the rigid limitations of traditional static rules; the three-level intelligent supplementation mechanism combines cross-source semantic association and knowledge graph technology to achieve accurate completion when data is missing, ensuring data continuity and availability without relying on manual intervention.

[0017] 3. The five-layer decoupled architecture of this invention achieves graded asset accumulation from raw data to decision data through "semantic anchors," "quality watermarks," and tracking mechanisms, significantly improving data value density. Furthermore, the standardized data product package and asset certificate design support fine-grained data reuse, end-to-end traceability, and commercial transactions, meeting the needs of agile and customized data services.

[0018] 4. The inter-layer linkage of this invention is achieved through event-driven and contractual schema, so that adding new data sources does not require modification of multi-layer code, greatly improving scalability; the optimized storage format and pre-aggregation design save storage space while significantly improving query response speed, and combined with the end-to-end quality monitoring mechanism, ensures efficient and reliable data processing. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the five-layer decoupling architecture and inter-layer linkage mechanism of the present invention; Figure 2 A schematic diagram of the LLM enhanced cleaning process and three-level prompting strategy; Figure 3 This is a schematic diagram of the dynamic semantic filtering and intelligent fill-in decision-making process. Detailed Implementation

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments, but this should not be construed as limiting the present invention.

[0021] Example: A method for integrating and storing multi-source data on the international communication influence of cities. This method is based on a five-layer decoupled architecture with native embedding of a Large Language Model (LLM). The five-layer decoupled architecture, from bottom to top, consists of a source buffer layer (SBL), a semantic enhancement operation layer (SE-ODS), a topic model layer (CDM), an intelligent marketplace layer (ADM), and a service assetization layer (DAS). The core of the five-layer decoupled architecture depends on big data processing components, an AI model deployment environment, and a data storage system. The following environment and component preparations need to be completed before deployment: The data acquisition component uses Airflow 2.6+, deployed in a cluster mode with at least 3 nodes, supporting DAG parallel scheduling for scheduled acquisition of multi-source data, task orchestration, and end-to-end monitoring. The storage system includes HDFS 3.x and Redis 7.x, with HDFS storage capacity at least 10TB for storing structured and unstructured data at various levels; the Redis cluster cache capacity is at least 500GB for storing semantic cache and hot data. The computing framework uses Spark 3.3+ and Flink 1.16+, with Spark deployed in YARN mode, each node configured with at least 16 cores and 64GB of resources, responsible for batch data cleaning and aggregation calculations; Flink is used for real-time data processing.

[0022] The model environment requires an NVIDIA GPU, with options for a consumer-grade RTX 4090 or an enterprise-grade A100, with at least 24GB and 80GB of VRAM respectively. CUDA 12.0+ must be installed to deploy the fine-tuned LLM model, supporting semantic processing and inference tasks. Metadata management utilizes Apache Atlas 2.3+ and a Schema Registry. The former supports field-level lineage tracing, while the latter manages schema versions, ensuring data lineage records, quality metric transmission, and schema compatibility checks. The service framework uses FastAPI 0.100+, deployed on a Kubernetes cluster with load balancing, to provide RESTful API data services and encapsulate various data product packages.

[0023] like Figure 1 As shown, the method includes the following steps: S1: Collect multi-source heterogeneous data and store it in the source buffer layer. The multi-source heterogeneous data includes social media data, news data, search index data, and travel review data. When storing, the original data and its collected metadata are retained and organized using a partitioned directory. In this step, data collection is configured using Airflow DAG orchestration to collect tasks, with multi-source data collection starting in parallel daily at 02:00 UTC, covering four core data sources. Social media data is collected via Facebook / YouTube open APIs, including posts, comments, and interactions over the past 24 hours. API request frequency strictly adheres to platform rate limiting rules, while recording metadata such as remaining request counts and collection IPs. News data is obtained from LexisNexis data interfaces, acquiring city-related reports from major global news portals, including fields such as title, body text, publication time, and media level, in XML format. Search index data is collected via Google Trends API, including keyword search popularity, geographical distribution, and time trends for target cities, in CSV format. Travel review data is collected via TripAdvisor and Booking.com open interfaces, including reviews of hotels, attractions, and restaurants, covering fields such as review text, ratings, publication time, and user profiles.

[0024] The source buffer layer adopts a dual-schema storage design of "raw data and collected metadata". The raw data maintains the original format of the data source. JSON nested data, XML text, etc. are all stored in full as strings without any field parsing to avoid information loss due to premature structuring. The collected metadata adds 18 fields to each piece of raw data. The core fields include collection timestamp, API version, data freshness TTL, data source type, collection batch number, data integrity flag, etc., comprehensively recording relevant information about data collection.

[0025] The data storage adopts a three-level partitioned directory structure, organized and stored in HDFS according to "data source type, collection batch, and time". The time partition granularity is accurate to the hour, which facilitates precise traceability of data lineage. The storage format uses Parquet columnar storage combined with ZSTD compression, which can save 60% of storage space compared to the traditional TextFile format, and supports predicate pushdown optimization, which can significantly improve the processing efficiency of subsequent data cleaning stages.

[0026] The data lineage initialization process registers SBL layer data assets through Apache Atlas, recording in detail the acquisition task ID, source interface address, storage path, and metadata field relationships for each data file, thus constructing the initial link of the data lineage and laying the foundation for subsequent data traceability and quality issue localization.

[0027] S2: Push the data from the source buffer layer to the semantic enhancement operation layer, use a large language model to clean and semantically annotate the data, and generate a semantically rich record containing semantic annotation fields and quality confidence fields; In this step, model loading and warm-up are performed after the semantic enhancement operation layer processing task starts at 04:00 UTC daily. First, the Qwen2.5-72B model, which has been domain-adaptively fine-tuned, is loaded. Model deployment uses GPTQ4-bit quantization technology to compress the model size to 36GB, which can be deployed on RTX4090 / A100 GPUs, with inference latency controlled within 800ms. Cache warm-up loads the semantic fingerprints (SimHash values) of the past 7 days and the corresponding LLM processing results from the Redis semantic cache library to build an in-memory SimHash query tree, supporting fast matching of new data and cached data.

[0028] like Figure 2 As shown, in the semantic fingerprint calculation and cache matching process, for each piece of data passed from the source buffer layer, the 128-bit SimHash value of its key fields (text content, core entity-related fields) is first calculated as the semantic fingerprint. Then, the Hamming distance between the semantic fingerprint of the new data and the Hamming distance recorded in the cache is calculated. If the distance is less than 5, the LLM processing result in the cache is directly reused, and the model call is skipped; if there is a miss or the distance is not less than 5, the three-level hint processing flow is entered.

[0029] The hybrid prompting engineering strategy processes text based on its complexity and value. For conventional data (90%) shorter than 100 characters, without multilingual content or special symbols, L1-level hard prompts are used, employing a fixed template: "Please identify the sentiment (positive / negative / neutral), entity information (city, attraction, etc.), and whether it is advertising / spam. Output only the label and confidence level: {text}". The response time is approximately 300ms per message, completing basic semantic processing at low cost. For complex data (8%) longer than 200 characters, with multilingual content or dialects, L2-level few-shot prompts are used, embedding three domain-similar examples in the prompts. For example, when processing Arabic dialect comments, the prompts include "original dialect text,..." The standard Arabic translation and semantic annotation examples guide the model to understand the context and improve the accuracy of semantic recognition. For high-value data such as posts from influential figures with more than 100,000 followers and texts related to trending events (accounting for 2%), L3-level thinking chain prompts are used, employing a step-by-step reasoning template: "Step 1: Identify the city and related entities (attractions, policies, etc.) in the text; Step 2: Determine the sentiment (including mixed sentiment analysis); Step 3: Identify potential risks (political sensitivity, false information, etc.); Step 4: Output structured JSON format, including the original text, translation results, annotations for each dimension, and confidence level." The response time is approximately 2 seconds per data point, enabling deep semantic information mining.

[0030] like Figure 3 As shown, dynamic semantic filtering performs semantic entropy monitoring and threshold adjustment on each batch of 1000 data entries. Semantic entropy is based on the formula... Calculation, where (This refers to the TF-IDF weights of each word in the text). If the semantic entropy deviates from the historical baseline by more than 20%, it is determined to be a data distribution drift (such as a sudden event causing topic concentration), and is calculated using the formula... Automatically adjust the noise detection threshold, where, This is the sensitivity coefficient, with a default value of 0.5. (This is the entropy offset). Simultaneously, context-aware filtering is used to lower the filtering threshold for text containing city entities and raise the threshold for purely forwarded text without substantial content, ensuring that core information is not lost.

[0031] After the rich semantic records are generated, each data entry contains a four-tuple structured field. The original fields retain the original text from the SBL layer and the data source platform information; the standard fields include machine-translated unified language text (default Chinese / English) and a processing timestamp; the semantic annotation fields cover sentiment and confidence levels, entity recognition results, and noise markers; the quality confidence fields include a comprehensive quality score (0-1) and a 128-bit SimHash semantic fingerprint. Data is partitioned by "data source type_processing date_hour" and stored in the HDFS SE-ODS directory, still in Parquet+ZSTD format. Simultaneously, the semantic fingerprint and processing results are written to a Redis cache to support subsequent data association and deduplication.

[0032] Table 1 shows the quantitative comparison of the advantages of the present invention with the prior art: Table 1

[0033] As shown in Table 1, this layer achieves significant quantitative advantages over existing technologies through a three-pronged LLM application strategy of "domain-adaptive fine-tuning, hybrid prompting engineering, and semantic caching": In cross-language sentiment analysis accuracy, it far surpasses traditional keyword matching schemes and general LLM API call schemes; the cost of processing a single data point is significantly lower than that of general LLM API, remaining within the reasonable range of traditional keyword matching schemes; processing latency is significantly reduced, much faster than general LLM API calls, and slightly higher than traditional keyword matching while ensuring the effectiveness of deep semantic processing; noise recognition recall is also significantly better than traditional keyword matching and general LLM API calls, effectively solving the problem of insufficient semantic noise recognition capability in traditional schemes. These quantitative advantages fully verify the technological breakthroughs of this invention in semantic processing accuracy, cost control, and efficiency improvement, providing a high-quality, high-value, rich semantic data foundation for subsequent layers of data processing.

[0034] S3: Based on the semantically enhanced operation layer outputting the semantically rich records, construct a topic model containing detailed data and pre-aggregated summary data in the topic model layer, and add data quality watermarks to the data partitions; In this step, the Detailed Data Layer (DWD) modeling is performed after the CDM layer task starts at 06:00 UTC daily. The semantically enhanced operation layer's rich semantic data is distributed according to business themes, constructing three major detailed data tables. `dwd_post_detail` (post details) is partitioned by city ID and date, storing fields such as original post text, translated text, sentiment rating, interaction volume (likes / comments / shares), entity information, and quality rating. `dwd_review_detail` (review details) is partitioned by city ID, platform, and date, storing fields such as ratings, text content, sentiment bias, and noise markers related to travel reviews. `dwd_account_profile` (account profile) is partitioned by platform and account ID, storing fields such as number of followers, historical post count, average sentiment bias, and account type (individual / organization / influencer).

[0035] The Data Summary Layer (DWS) pre-aggregation cube is built based on detailed data from the DWD layer. It pre-calculates aggregation metrics for 50 commonly used dimension combinations, forming a pre-aggregation cube that significantly improves query response speed. Core pre-aggregation metrics include city daily / weekly / monthly sentiment averages, platform-level dissemination rankings, top 10 most popular tourist attractions reviews, and positive news reporting rate. Taking the city's monthly sentiment average pre-aggregation as an example, it is implemented using SQL statements: INSERT INTO dws_city_monthly_sentiment SELECT city_id, DATE_TRUNC('month', post_date) as month, AVG(sentiment_score) as avg_sentiment, COUNT(*) as post_count, QUALITY_WATERMARK(0.92, 0.88, 0.95, 0.98) as quality_qr_code FROM dwd_post_detail GROUP BY city_id, DATE_TRUNC('month', post_date).

[0036] The QUALITY_WATERMARK function generates a quality QR code that includes four scores: completeness, accuracy, consistency, and timeliness. This QR code is stored in association with the aggregated data and flows with the data.

[0037] The baseline calculation for the Influence Index is based on five primary indicators (online dissemination, media coverage, social media, information retrieval, and international visitors). Each primary indicator contains 3-5 secondary indicators, and standardized scores are calculated by extracting relevant data from the DWS layer. For example: Social media influence score = Normalized follower count × w1 + Normalized engagement rate × w2 + Normalized sentiment index × w3; Among them, w1-w3 are weights, which are set according to the number of language users / platform user volume. The Z-score standardization method is used to convert the original values ​​of each indicator into normalized values ​​in the range of [0,1] to ensure that indicators of different dimensions can be calculated in a weighted manner.

[0038] S4: Based on query hotspots, dynamically generate materialized views in the smart marketplace layer and perform cross-source intelligent fill-in when the data source is missing; In this step, hotspot query identification and materialized view construction are executed after the ADM layer task starts at 08:00 UTC daily. First, the DAS layer API call logs for the past 7 days are analyzed to identify high-frequency query patterns. If a query pattern (e.g., "city_id=123 AND date_range='last_7_days' AND theme='hotel_sentiment'") is called at least 50 times, it is identified as a hotspot query, and a corresponding materialized view is automatically created. The materialized view is incrementally updated every 2 hours to ensure data timeliness. For example, the SQL statement for creating a hotel sentiment hotspot materialized view is: CREATE MATERIALIZED VIEW adm_hotel_sentiment_hot AS SELECT city_id, hotel_name, AVG(sentiment) as sentiment_7d, COUNT(*) as FROM dwd_review_detail WHERE category='hotel' AND dt>= CURRENT_DATE - 7 GROUP BY city_id, hotel_name.

[0039] The intelligent data backup mechanism is activated when a data source is detected to be missing (such as TripAdvisor data for a certain city not being updated) or when the data delay exceeds 2 hours. It includes three levels of backup modes. Level 1 homogeneous historical data complementation is suitable for scenarios with data delays. It retrieves the historical data of the city and the data source for the most recent 7 days, and predicts the values ​​for the missing period using a time series model (such as ARIMA). The complementation confidence level is 0.8-0.9. Level 2 cross-source semantic data complementation is suitable for scenarios where the data source is completely missing. It retrieves records with a semantic similarity of no less than 0.85 from other platforms (such as Booking.com and Google Reviews) using semantic fingerprints from the SE-ODS layer (similarity is calculated using multilingual Sentence-BERT), extracts core information such as sentiment and keywords as complementation data, and adds a "cross_source_complement_from_xxx" source tag. Level 3 knowledge graph data complementation is suitable for scenarios where multiple sources of data are missing. Based on the city, entity, and sentiment knowledge graph, it queries the historical sentiment baseline of the city (such as the average sentiment score of tourism texts), combines it with current hot events (such as exhibitions and events) to adjust the sentiment shift, and generates synthetic labeled data. The complementation confidence level is 0.5-0.6, and it is explicitly marked "complementation generated".

[0040] The indicator calculation process breaks down the calculation of influence score into three stages: normalization, weighting, and summation. These stages are stored in three tables: adm_normalization (normalized value), adm_weighted (weighted value), and adm_final_score (final score). This allows users to trace the composition of the score, improves data transparency, and facilitates subsequent analysis and verification.

[0041] S5: In the service assetization layer, data at each level is encapsulated into data product packages of different granularities, and corresponding data service interfaces are provided.

[0042] In this step, the data product package is encapsulated daily at 10:00 UTC, encapsulating data at each level into four standard data product packages. The raw data package contains raw SBL layer data and collected metadata, granularized by "city ID and date," supporting raw data backtracking; the semantic annotation package contains semantically rich records from the SE-ODS layer, with added LLM processing confidence and quality scores, suitable for deep semantic analysis scenarios; the topic dataset contains detailed DWD data and pre-aggregated DWS data from the CDM layer, granularized by "topic and city ID," with added quality watermarks, supporting business topic analysis; and the thematic report package contains aggregated data and supplementary data from the ADM layer, supporting export in JSON, PDF, and Excel formats, suitable for decision-making reporting scenarios.

[0043] Data asset certificate generation equips each data product package with a digital asset certificate, which includes four core modules: Data lineage record: This records the complete processing path from the raw data at the SBL layer to the product package at the DAS layer (including the processing task ID and model version at each layer); Quality score details: This displays the four scores for completeness, accuracy, consistency, and timeliness, along with their calculation basis; Version information: This includes the data processing batch, model version, and schema version; Usage license and billing: This clarifies the data's applicable scope (e.g., "internal analysis," "commercial license"), update frequency (e.g., "daily update," "weekly update"), and billing model (e.g., "billing per thousand records," "billing per access").

[0044] Service publishing utilizes FastAPI to publish RESTful API interfaces, providing five core data services. The raw data interface " / api / v1 / raw / {city_id} / {date}" returns raw SBL layer data for a specified date in the corresponding city; the semantic annotation interface " / api / v1 / semantic / {city_id} / {date}" returns rich semantic data from the SE-ODS layer; the topic data interface " / api / v1 / theme / {theme_name} / {city_id}" returns CDM layer data for a specified topic and city; the special report interface " / api / v1 / report / {city_id} / {period}" returns a special report from the ADM layer for a specified city and time period; and the complement data interface " / api / v1 / complement / {city_id} / {source}" returns complement data for a specified city and data source, supporting filtering by confidence level.

[0045] The entire process is monitored and traced back through Airflow, which monitors the execution status of all tasks and records the execution time, CPU / memory consumption, data input / output volume, quality score changes, and number of retries for each task. The system supports task-level reruns and data backtracking. When a quality problem occurs at a certain level of data, the source of the problem can be located through the lineage, and the corresponding task can be rerun to generate new data, ensuring the reliability and accuracy of data processing.

[0046] In this embodiment, an instruction dataset for urban communication is constructed, containing 50,000 labeled samples (covering sentiment recognition, entity extraction, and noise classification). The Qwen2.5-72B model is fine-tuned using LoRA low-rank adaptation technology, training only 0.1% of the parameters (approximately 70M). Fine-tuning can be completed in 3 hours on a single A100 sheet, ensuring that the model is suitable for urban international communication scenarios.

[0047] During initial deployment, the semantic entropy baseline (H_base) of each data source needs to be calculated based on 30 days of historical data, and the sensitivity coefficient (a) is adjusted according to the data type. For example, a=0.6 for social media data (more sensitive to drift) and a=0.4 for news data (higher stability).

[0048] Set a quality score threshold (default ≥0.7). Data below this threshold will be marked as "low quality data" and a notification will be sent when the API returns the data. At the same time, when the average quality score of a batch of data is <0.6, an alarm will be triggered and the corresponding processing task will be automatically rerun.

[0049] When adding a new data source, you only need to configure the collection task in the SBL layer and register the data source schema in the Schema Registry, without modifying other layers of code; through the event-driven mechanism (EventBus), the new data source data automatically triggers the subsequent processing flow of each layer, achieving architectural decoupling.

Claims

1. A method for integrating and storing structured data on a city's international communication influence from multiple sources, characterized in that: This method is based on a five-layer decoupled architecture with native embedding of a large language model. The five-layer decoupled architecture, from bottom to top, consists of a source-attachment buffer layer, a semantic enhancement operation layer, a topic model layer, an intelligent marketplace layer, and a service assetization layer. The method includes the following steps: S1: Collect multi-source heterogeneous data and store it in the source buffer layer. The multi-source heterogeneous data includes social media data, news data, search index data, and travel review data. When storing, the original data and its collected metadata are retained and organized using a partitioned directory. S2: Push the data from the source buffer layer to the semantic enhancement operation layer, use a large language model to clean and semantically annotate the data, and generate a semantically rich record containing semantic annotation fields and quality confidence fields; S3: Based on the semantically enhanced operation layer outputting the semantically rich records, construct a topic model containing detailed data and pre-aggregated summary data in the topic model layer, and add data quality watermarks to the data partitions; S4: Based on query hotspots, dynamically generate materialized views in the smart marketplace layer and perform cross-source intelligent fill-in when the data source is missing; S5: In the service assetization layer, data at each level is encapsulated into data product packages of different granularities, and corresponding data service interfaces are provided.

2. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 1, characterized in that: In step S1, the source buffer layer uses a dual schema storage of raw data and collected metadata. The collected metadata includes the collection timestamp, API version, and data freshness TTL. The raw data is stored in three partitions according to the data source type, collection batch, and time. The partitioned directory adopts a three-level structure of data source type, collection batch, and collection time, with partition granularity accurate to the hour.

3. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 1, characterized in that: Step S2, which involves cleaning and semantic annotation using a large language model, specifically includes: Use a large language model that has been domain-adaptively fine-tuned as the base model; A hybrid cueing engineering strategy is adopted, which selectively applies hard cues, few-sample cues, or thought chain cues based on the complexity and value of the text. A semantic caching library is built. A semantic fingerprint is calculated for new text. If the Hamming distance between the new text and the semantic fingerprint recorded in the cache is less than a set threshold, the cached result is directly reused.

4. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 3, characterized in that, The hybrid prompting engineering strategy includes: For regular data, hard hints are provided using a fixed template without contextual examples. For complex text, a few-sample hint containing examples from several domains is used for processing; For high-value texts, a thought chain prompting method that requires step-by-step reasoning using a large language model is employed.

5. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 1, characterized in that, The semantically rich record generated in step S2 contains a four-tuple structure: original field, standard field, semantic annotation field, and quality confidence field; the semantic annotation field includes at least machine-translated text, sentiment score, entity recognition result, and noise label; and a 128-bit semantic fingerprint is generated for each record for deduplication and association.

6. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 1, characterized in that, Step S2 also includes a dynamic semantic filtering step: Calculate the semantic entropy value for each batch of data; Monitor the offset of the semantic entropy value relative to the historical baseline; When the offset exceeds the preset threshold, the noise judgment threshold in the data cleaning process is automatically adjusted.

7. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 1, characterized in that, In step S3, the topic model layer includes a detailed data layer and a summary data layer. The detailed data layer is partitioned by time, city, and topic. The summary data layer stores pre-calculated aggregated indicator cubes.

8. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 1, characterized in that, The intelligent fill-in mechanism described in step S4 includes at least one of the following mechanisms: Homologous historical fill: Predicting missing values ​​based on time series data from historical sources; Cross-source semantic padding: Filling in records with semantic similarity by retrieving semantic fingerprints from other data sources; Knowledge graph supplementation: Generate synthetic labeled data based on the historical sentiment baseline of the city entity knowledge graph and current hot events.

9. The method for integrating and structured storage of multi-source data on urban international communication influence according to claim 1, characterized in that, In step S5, the packaged data product package includes a data asset certificate, which contains data lineage information, quality score details, processing version information, and usage license information.

10. The method for integrating and structured storing multi-source data on the international communication influence of cities according to claim 1, characterized in that, The five-layer decoupled architecture links the layers through an event-driven mechanism, and the data flow between layers transmits quality scoring information through metadata services, forming a quality impact chain.

Images