A marketing data-oriented multi-source heterogeneous intelligent cleaning and standardization processing method and system

Abstract

The application discloses a multi-source heterogeneous intelligent cleaning and standardization processing method and system for marketing data. The system comprises a data access and adaptation module for converting multi-source heterogeneous data into unified intermediate representation layer data and completing preliminary extraction and structuring of unstructured data; a data quality detection and monitoring module for automatically identifying data quality problems and realizing real-time monitoring and early warning; an intelligent data cleaning module for adaptively cleaning data quality problems by using a double-engine architecture and synchronously executing marketing data privacy compliance desensitization processing; a standardization processing and mapping module for completing data mapping and conversion based on unified marketing data standards; and a quantitative data quality evaluation and feedback module for completing data quality quantitative scoring and forming a feedback closed loop. The problems of low data quality, insufficient standardization degree, processing link fault and lack of continuous optimization closed loop in traditional marketing data processing are effectively solved.

Description

Technical Field

[0001] This application relates to the field of marketing data processing technology, and more specifically, to a method and system for multi-source heterogeneous intelligent cleaning and standardization processing of marketing data. Background Technology

[0002] Currently, enterprises face the common challenge of low data quality and insufficient standardization when using multi-source, heterogeneous marketing data for decision-making. Existing technical solutions are typically pieced together from multiple independent, loosely coupled tools or processes, resulting in gaps in the processing steps and making it difficult to achieve fully automated workflow from data access, quality discovery, cleaning and correction to standardized output. Particularly for unstructured data such as text and images, the lack of effective technical means to deeply understand their content and identify potential quality issues significantly reduces the credibility and value of subsequent analysis. Summary of the Invention

[0003] The main objective of this application is to provide a method and system for multi-source heterogeneous intelligent cleaning and standardization of marketing data, aiming to solve at least one of the above-mentioned technical problems.

[0004] The first aspect of this application provides a multi-source heterogeneous intelligent cleaning and standardization processing system for marketing data, including: The data access and adaptation module is used to convert multi-source heterogeneous data into unified intermediate representation layer data, and to complete the initial extraction and structuring of unstructured data. The data quality detection and monitoring module is used to automatically identify data quality problems and achieve real-time monitoring and early warning. The intelligent data cleaning module is used to adaptively clean data with a dual-engine architecture to address data quality issues, and simultaneously performs privacy compliance de-identification processing on marketing data. The standardization processing and mapping module is used to complete data mapping and transformation based on a unified marketing data standard. The quantitative data quality assessment and feedback module is used to complete the quantitative scoring of data quality and form a feedback loop. The data quality detection and monitoring module includes: A dynamic rules engine is used to execute configurable business rules and achieve adaptive threshold adjustment; The machine learning detection unit is used to identify outliers and isolated points using unsupervised learning algorithms and to discover logical errors using sequence pattern mining. Deep quality analysis models are used to learn semantic features and identify quality problems in unstructured data using deep learning models. A visual monitoring and early warning platform is used to display data quality indicators and implement multi-level early warnings; The deep quality analysis model includes a semantic inconsistency detection unit, which performs hierarchical semantic representation on unstructured text data and calculates its bidirectional consistency constraint loss through a trained reconstruction network to identify semantic inconsistencies.

[0005] In some embodiments of this application, the data access and adaptation module includes a multimodal adapter framework, a metadata-driven management unit, and a unified data interface layer; The multimodal adapter framework is used to develop dedicated adapters for different data sources and data formats, converting data into unified intermediate representation layer data; The metadata-driven management unit is used to centrally manage the metadata information of the access data source; The unified data interface layer is used to provide a standardized data access interface and connect to the unstructured data parsing engine to achieve information extraction and structuring.

[0006] In some embodiments of this application, the unstructured data parsing engine includes an OCR parsing unit and an NLP parsing unit, and the unstructured data parsing engine is used to perform preliminary extraction and structuring of text and images.

[0007] In some embodiments of this application, the deep quality analysis model further includes an emotional conflict detection unit, which is used to extract emotional features from text, speech and image modal data associated with the same data object, map them to a unified emotional space through cross-modal alignment and calculate the difference degree to determine emotional conflict-type quality problems.

[0008] In some embodiments of this application, the deep quality analysis model further includes a marketing intent feature extraction and association unit, which is used to extract marketing intent features of data through a multimodal fusion network with a denoising bottleneck, and associate and label the quality problems of semantic inconsistency and emotional conflict with specific marketing intent features based on preset mapping rules. The semantic inconsistency detection unit, the sentiment conflict detection unit, and the marketing intent feature extraction and association unit share the underlying feature encoder and are trained collaboratively through multi-task joint loss to simultaneously complete the identification of quality problems and the association of intent features.

[0009] In some embodiments of this application, the intelligent data cleaning module includes a rule-driven cleaning engine, an artificial intelligence-driven cleaning engine, a cleaning strategy optimizer, a privacy compliance desensitization unit, and an edge lightweight cleaning acceleration unit. The rule-driven cleaning engine is used to perform standardized cleaning operations based on the general rule layer and the marketing business rule layer; The AI-driven cleaning engine is used to complete missing value filling, outlier correction, duplicate record matching and merging, and semantic-level deep cleaning. The cleaning strategy optimizer is used to iteratively optimize the artificial intelligence model and the cleaning rule base based on cleaning feedback data. The privacy compliance desensitization unit is used to perform marketing data desensitization processing in conjunction with the cleansing process; The edge lightweight cleaning acceleration unit is used to perform real-time lightweight cleaning calculations on the edge side.

[0010] In some embodiments of this application, the AI-driven cleaning engine includes a missing value intelligent filling unit, an outlier identification and correction unit, a duplicate record fuzzy matching and merging unit, a semantic-level deep cleaning unit, and a marketing intent association correction unit; The semantic-level deep cleaning unit is used to combine pre-trained language models and domain knowledge graphs to complete entity linking, relationship verification and semantic error correction. The marketing intent association correction unit is used to perform targeted correction of quality issues based on the extracted marketing intent features; The duplicate record fuzzy matching and merging unit is used to identify approximately duplicate items using an NLP similarity algorithm and perform a merging operation.

[0011] In some embodiments of this application, the standardization processing and mapping module includes a marketing data standard library, an intelligent mapping engine, and a federated cleaning and alignment unit; The aforementioned marketing data standard library is used to define a unified standard for enterprise-level marketing data. The intelligent mapping engine is used to automatically map and convert source data fields to standard fields based on metadata and semantic analysis; The federated cleaning and alignment unit is used to clean and standardize cross-platform marketing data in federated learning mode.

[0012] In some embodiments of this application, the marketing data standard library covers data formats, encoding rules, units of measurement, business terminology, customer classification systems, and product classification system standards.

[0013] Another aspect of this application provides a method for intelligent cleaning and standardization of multi-source heterogeneous marketing data, including: Unified access and adaptation of multi-source heterogeneous marketing data; conversion of multi-source heterogeneous marketing data into unified intermediate representation layer data; and preliminary extraction and structuring of unstructured marketing data. The system performs data quality detection and monitoring on marketing data after unified access and adaptation, automatically identifies data quality problems, extracts marketing intent features, and performs real-time monitoring and multi-level early warning. It adopts a dual-engine architecture that integrates rule-driven and AI-driven approaches to intelligently clean quality issues identified by quality detection, while simultaneously performing marketing data privacy compliance desensitization and real-time edge-side cleaning acceleration. Based on the preset unified standard for marketing data, the marketing data after intelligent cleaning is subjected to standardized processing and field mapping transformation, and cross-platform data cleaning and alignment is completed in federated learning mode. The standardized marketing data is subjected to multi-dimensional quantitative quality assessment, and the assessment results are fed back to the data quality detection, intelligent data cleaning and standardization process to form a closed loop for quality optimization.

[0014] The embodiments of this application include at least the following beneficial effects: Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of this application, the relevant drawings of the embodiments of this application are described below. It should be understood that the drawings described below are only for the convenience of clearly describing some embodiments of the technical solutions of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of a multi-source heterogeneous intelligent cleaning and standardization processing system for marketing data proposed in an embodiment of this application; Figure 2 This is a schematic diagram of the data access and adaptation module proposed in an embodiment of this application; Figure 3 This is a schematic diagram of the data quality detection and monitoring module proposed in an embodiment of this application; Figure 4 This is a schematic diagram of the intelligent data cleaning module proposed in an embodiment of this application; Figure 5 This is a schematic diagram of the artificial intelligence-driven cleaning engine proposed in an embodiment of this application; Figure 6 This is a flowchart of a multi-source heterogeneous intelligent cleaning and standardization processing method for marketing data proposed in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application.

[0018] Traditional marketing data processing solutions generally suffer from low data quality and insufficient standardization when using multi-source heterogeneous data for decision-making. Existing solutions are often pieced together from independent tools, resulting in gaps in the processing flow and making it difficult to achieve fully automated workflows from data access to standardized output. Particularly with unstructured data, there is a lack of effective means to identify potential quality issues, and quality management lacks continuous feedback and optimization loops, making it difficult to continuously improve data quality.

[0019] In response, the first aspect of this application proposes a multi-source heterogeneous intelligent cleaning and standardization processing system for marketing data. See also... Figure 1 The system includes: a data access and adaptation module, used to convert multi-source heterogeneous data into unified intermediate representation layer data and complete the initial extraction and structuring of unstructured data; a data quality detection and monitoring module, used to automatically identify data quality problems and realize real-time monitoring and early warning; an intelligent data cleaning module, used to adaptively clean data quality problems using a dual-engine architecture, and simultaneously perform marketing data privacy compliance desensitization processing; a standardization processing and mapping module, used to complete data mapping and transformation based on unified marketing data standards; and a quantitative data quality assessment and feedback module, used to complete data quality quantitative scoring and form a feedback closed loop.

[0020] For ease of understanding, the following explains some key terms in this embodiment: Multi-source heterogeneous data refers to data collections originating from different systems, formats, and structures. Examples include customer information, transaction records, behavioral data, text comments, and images from CRM systems, ERP systems, social media platforms, website logs, email marketing, and other channels. These data differ in type, storage method, and semantics.

[0021] Unified intermediate representation layer data refers to the process of converting the aforementioned multi-source heterogeneous data into an intermediate data form with a unified format, structure, and semantics after preliminary processing. This intermediate representation layer data aims to eliminate the heterogeneity of the original data and provide a consistent foundation for subsequent data quality detection, cleaning, and standardization processes.

[0022] Unstructured data refers to data that does not have a predefined data model or a fixed format, such as text, images, audio, and video. In marketing data, common unstructured data includes customer reviews, social media posts, email content, and product images.

[0023] A dual-engine architecture refers to the simultaneous use of two different types or focuses of cleaning mechanisms in the intelligent data cleaning process. For example, it can combine a cleaning engine based on preset rules with a cleaning engine based on an artificial intelligence model to achieve more comprehensive and flexible data cleaning capabilities.

[0024] Privacy-compliant desensitization refers to the process of anonymizing, pseudonyming, generalizing, or encrypting marketing data containing personal identification information or other sensitive information in accordance with relevant laws, regulations, and privacy protection policies during data processing. This aims to reduce the risk of data leakage and ensure the legality and security of data use.

[0025] A unified marketing data standard refers to a set of specifications developed within an enterprise or industry to ensure the consistency, comparability, and interoperability of marketing data. This standard typically covers data field definitions, data types, coding rules, units of measurement, business terminology, customer classification systems, and product classification system standards.

[0026] Feedback loop refers to the process of tracing the final evaluation results or processing effectiveness information back to the beginning or middle of the data processing workflow. This feedback guides and optimizes subsequent processing strategies or parameter adjustments. Through continuous feedback and adjustments, continuous improvement in data quality and iterative optimization of system performance are achieved.

[0027] As one implementation method, the data access and adaptation module is configured to convert multi-source heterogeneous data into unified intermediate representation layer data and perform preliminary extraction and structuring of unstructured data. Specifically, this module can receive data from different databases, file systems, or API interfaces. For example, for structured data, format conversion can be performed using preset connectors or scripts; for unstructured data, key information can be extracted from text or documents using keyword matching or simple template recognition, and organized into a preliminary structured form.

[0028] The data quality detection and monitoring module is configured to automatically identify data quality issues and provide real-time monitoring and early warnings. Based on a pre-defined set of static rules, this module scans the incoming unified intermediate presentation layer data to detect data format errors, missing values, or outliers exceeding preset ranges. When a quality issue is detected, an alert can be issued via simple log recording or email notification.

[0029] The intelligent data cleaning module is configured with a dual-engine architecture to adaptively clean data to address quality issues, while simultaneously performing privacy-compliant anonymization of marketing data. One cleaning engine operates based on a set of fixed cleaning rules, such as deleting duplicate records or correcting common spelling errors. The other cleaning engine uses a statistical model to perform simple outlier smoothing or missing value imputation. Privacy-compliant anonymization can be performed on sensitive fields through manual review or simple replacement rules.

[0030] The standardization processing and mapping module is configured to perform data mapping and transformation based on a unified marketing data standard. This module can maintain a static mapping table containing standard field definitions and transformation rules. When cleaned data is received, the module will manually or through simple lookup matching convert the source data fields into standard fields according to this mapping table, and adjust the data format to conform to the unified standard.

[0031] The quantitative data quality assessment and feedback module is configured to complete quantitative data quality scoring and form a feedback loop. This module can score processed data based on a preset single-dimensional indicator, such as data completeness rate. The scoring results can generate reports periodically, and after manual analysis, provide written feedback to data access, cleaning, or standards development personnel to guide subsequent optimization efforts.

[0032] The system proposed in this application effectively solves the problems of low data quality, insufficient standardization, fragmented processing, and lack of continuous optimization loop in traditional marketing data processing by integrating modules such as data access and adaptation, quality detection and monitoring, intelligent cleaning, standardized processing, and quantitative evaluation and feedback. This system enables fully automated processing of multi-source heterogeneous data, improves the understanding and quality identification capabilities of unstructured data content, and ensures that data quality continuously and automatically improves along with business development, thereby providing more reliable and higher-value data support for enterprise marketing decisions.

[0033] See Figure 2 In some embodiments of this application, the data access and adaptation module includes a multimodal adapter framework, a metadata-driven management unit, and a unified data interface layer.

[0034] The multimodal adapter framework is a flexible and scalable architecture whose main function is to develop dedicated adapters for different types of data sources (e.g., internal enterprise databases, third-party platform APIs, file systems, etc.) and various data formats (e.g., structured CSV, JSON, XML, relational database tables, and semi-structured or unstructured text, images, etc.). Each adapter is responsible for parsing its specific data source and format, and converting its content into predefined, unified intermediate representation layer data within the system. This modular design allows the system to easily cope with constantly changing data sources and formats without requiring large-scale modifications to the core system, thus ensuring broad compatibility and high scalability of data access.

[0035] The metadata-driven management unit is used for centralized management of metadata information from access data sources. This metadata includes, but is not limited to, the data source type, location, access credentials, data schema, field definitions, data update frequency, data quality rules, and data transformation rules. By centrally managing this metadata, the system can dynamically configure and control the data access process. For example, it can automatically adjust data parsing and transformation strategies based on metadata, or quickly update processing logic when data changes. This driven management approach significantly improves the flexibility, maintainability, and automation of data access, and provides accurate and consistent contextual information for subsequent data processing.

[0036] The unified data interface layer provides a standardized data access interface and interfaces with unstructured data parsing engines to achieve information extraction and structuring. This interface layer provides a unified, abstract data access view for other modules within the system, shielding them from the complexity of underlying data sources and formats. When the adapter identifies unstructured data, the unified data interface layer routes this data to a specialized unstructured data parsing engine (e.g., an Optical Character Recognition (OCR) engine or a Natural Language Processing (NLP) engine) to perform preliminary extraction and structuring of key information from text, images, and other data. The processed structured data is then integrated back into the unified intermediate representation layer, ensuring that all types of data can be used by subsequent modules in a standardized format.

[0037] By introducing a multimodal adapter framework, a metadata-driven management unit, and a unified data interface layer, this system can efficiently and flexibly address the challenges of accessing multi-source heterogeneous marketing data. The multimodal adapter framework, through its modular design, ensures broad compatibility and scalability across various data sources and formats, avoiding access barriers caused by data source diversity. The metadata-driven management unit enables centralized and dynamic management of data source metadata, greatly improving the flexibility and maintainability of the data access process and providing accurate contextual information for subsequent data processing. The unified data interface layer not only provides standardized data access methods but also ensures, through seamless integration with the unstructured data parsing engine, that unstructured data such as text and images can be effectively and accurately extracted and structured, thus laying a solid data foundation for subsequent intelligent cleaning and standardization processing, significantly improving the efficiency and quality of data access.

[0038] In some embodiments of this application, the unstructured data parsing engine includes an OCR parsing unit and an NLP parsing unit. The unstructured data parsing engine is used to perform preliminary extraction and structuring of text and image data. Specifically, the OCR parsing unit's main function is to recognize and extract text information from images or scanned documents. Specifically, this unit can process unstructured data in image form, such as pictures and PDFs. It preprocesses the input image using a series of image processing algorithms (e.g., denoising, binarization, and tilt correction), then uses character segmentation technology to separate the text regions in the image and extracts features from each character. Finally, a pattern recognition algorithm maps these character features to corresponding text characters, thereby converting the image-based text content into machine-readable and editable text data. For example, in marketing scenarios, for images containing product information, promotional posters, or customer feedback screenshots, the OCR parsing unit can accurately extract the text information from the image, providing a data foundation for subsequent text analysis.

[0039] The NLP parsing unit's primary function is to perform deep understanding and information extraction from natural language text. Specifically, this unit can perform operations such as word segmentation, part-of-speech tagging, named entity recognition (e.g., identifying product names, brands, personal names, locations), sentiment analysis, topic modeling, and syntactic analysis on text data. Through these processes, the NLP parsing unit can identify key entities, the relationships between them, and the intent or sentiment expressed in the text from unstructured text. For example, for customer comments on social media or discussions in online forums, the NLP parsing unit can identify product features mentioned by customers, their satisfaction with the product, and potential purchase intentions, structuring this information into tags or attributes for further system processing.

[0040] Through the collaborative work of the OCR and NLP parsing units, the unstructured data parsing engine can comprehensively achieve preliminary extraction and structuring of text and image data. Specifically, for image and PDF data, the OCR parsing unit first accurately identifies and extracts the text information into plain text format; subsequently, both the original text data and the text data extracted by OCR are handed over to the NLP parsing unit for in-depth semantic analysis and information extraction. In this way, the originally scattered and disordered unstructured information is transformed into structured data with clear fields and values. For example, the phrase "A customer posted a picture on a social media platform with the text 'New product A, discounted price 199 yuan' and the caption 'This product A is really great, highly recommended!'" is initially structured as "Product Name: A; Price: 199 yuan; Customer Review: Great, highly recommended; Sentiment: Positive". This preliminary extraction and structuring greatly improves data usability and lays a solid foundation for subsequent data quality detection, intelligent cleaning, and standardization.

[0041] By introducing an OCR parsing unit and an NLP parsing unit into the unstructured data parsing engine, this application effectively solves the problems of low extraction efficiency and poor accuracy in traditional unstructured data processing when facing diverse text and image data in marketing scenarios. The OCR parsing unit can accurately identify and extract text information from image carriers such as pictures and PDFs, overcoming the obstacle of directly processing visual information. Meanwhile, the NLP parsing unit can perform deep semantic understanding and information extraction on these extracted texts and the original text, transforming ambiguous natural language into clear and meaningful structured data. This dual-unit collaborative approach enables the system to comprehensively, efficiently, and accurately complete the initial extraction and structuring of heterogeneous unstructured marketing data, significantly improving data quality and processing efficiency in the data access stage. It provides high-quality, highly available data input for subsequent data quality detection, intelligent cleaning, and standardization processing, thereby ensuring the robustness and effectiveness of the entire system when processing complex marketing data.

[0042] See Figure 3 In some embodiments of this application, the data quality detection and monitoring module includes a dynamic rule engine, a machine learning detection unit, a deep quality analysis model, and a visualization monitoring and early warning platform.

[0043] The dynamic rule engine executes configurable business rules and adaptively adjusts thresholds. This engine can pre-set a series of business rules for marketing data, such as data format validation, numerical range checks, and field correlation verification. These rules are configurable, allowing users to flexibly define and modify them according to actual business needs. Furthermore, the engine can adaptively adjust the trigger thresholds of rules based on historical data, the statistical characteristics of real-time data streams, or pre-set optimization strategies, thereby improving the accuracy and robustness of the rules and reducing false positives or false negatives. For example, regarding the fluctuation range of a key indicator, the system can automatically adjust its normal fluctuation threshold based on recent data trends to adapt to market changes.

[0044] The machine learning detection unit is used to identify outliers and anomalies using unsupervised learning algorithms and to discover logical errors using sequence pattern mining. This unit leverages machine learning techniques to detect anomalies in data that do not conform to conventional patterns. For example, through unsupervised learning algorithms such as Isolation Forest, One-Class SVM, or DBSCAN, the system can identify outliers or anomalies that are significantly different from the majority of data points. These outliers may represent data entry errors, system malfunctions, or fraudulent activities. Furthermore, through sequence pattern mining algorithms such as Apriori and PrefixSpan, this unit can analyze patterns in data sequences or event streams to discover errors that violate business logic. For example, in the customer lifecycle, the order in which certain events occur does not conform to the preset business process.

[0045] Deep quality analysis models employ deep learning to learn semantic features and identify quality issues in unstructured data. Given the large amount of unstructured information in marketing data, such as customer reviews, social media posts, and product descriptions, this model leverages the powerful capabilities of deep learning to perform in-depth analysis of this unstructured data. Specifically, the model can use deep learning architectures such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Transformers to learn high-level semantic feature representations from unstructured data such as text, images, and speech. Based on these semantic features, the model can identify quality issues that are difficult to detect using traditional methods, such as semantic inconsistencies, sentiment conflicts, missing information, or errors in text.

[0046] The visual monitoring and early warning platform is used to display data quality indicators and implement multi-level early warnings. The platform provides an intuitive user interface, visually displaying various quality issues, data quality scores, trend analyses, and other indicators identified by the data quality detection and monitoring module, allowing users to clearly understand the current data quality status. Simultaneously, the platform has multi-level early warning capabilities, triggering different levels of early warning notifications based on factors such as the severity and scope of the quality issues. These notifications can be sent via email, SMS, system messages, or dashboard highlighting, promptly notifying relevant personnel to take appropriate action.

[0047] By introducing a dynamic rule engine, machine learning detection units, deep quality analysis models, and a visual monitoring and early warning platform, this application comprehensively enhances the capabilities of the data quality detection and monitoring module. The dynamic rule engine can flexibly adjust thresholds according to business rules to adapt to constantly changing business needs; the machine learning detection unit effectively identifies outliers, discrepancies, and logical errors that are difficult to detect using traditional methods through unsupervised learning and sequence pattern mining; the deep quality analysis model utilizes deep learning technology to deeply understand the semantic features of unstructured data, thereby identifying deeper-level quality problems. Finally, the visual monitoring and early warning platform presents these detection results intuitively and provides multi-level early warnings to ensure that data quality problems are detected and addressed in a timely manner. This enables the system to more intelligently and comprehensively identify various quality problems in marketing data and provide real-time, accurate early warnings, significantly improving the efficiency and accuracy of data quality management and laying a solid foundation for subsequent cleaning and standardization processes.

[0048] In some embodiments of this application, the deep quality analysis model includes a semantic inconsistency detection unit for performing hierarchical semantic representation of unstructured text data and calculating its bidirectional consistency constraint loss through a trained reconstruction network to identify semantic inconsistencies.

[0049] Specifically, the semantic inconsistency detection unit is a dedicated component in the deep quality analysis model. Its core function is to identify and locate semantic contradictions or inconsistencies in unstructured text data. As an independent logical module, it receives preprocessed text data and outputs an evaluation result regarding its semantic consistency. To achieve this goal, the unit first performs hierarchical semantic representation on the unstructured text data. Hierarchical semantic representation refers to encoding unstructured text data at different granularities to capture its multi-layered semantic information. Specifically, advanced natural language processing techniques, such as pre-trained language models based on the Transformer architecture, can be used to convert text data into word vectors, sentence vectors, or paragraph vectors. These vectors can capture the contextual semantic information of words, sentences, and even entire text paragraphs. Hierarchical representation means that semantic features at the lexical, syntactic, and discourse levels can be considered simultaneously. For example, word embeddings can be used to obtain the semantics of words, sentence embeddings to obtain the semantics of sentences, and a higher-level encoder can be used to obtain the overall semantics of text blocks.

[0050] Building upon this, the system utilizes a trained reconstruction network. A reconstruction network is a neural network model whose primary task is to learn the inherent structure of input data and attempt to reconstruct it. In this implementation, the reconstruction network is designed and trained to reconstruct one semantic representation from another, or to reconstruct complete semantic information from partial semantic information. For example, it can be trained to reconstruct one semantic representation (e.g., a paragraph vector) of text from another semantic representation (e.g., a sentence vector), or to reconstruct the semantic representation of related text segments from the semantic representation of a text segment. During training, the network optimizes its parameters by minimizing the difference between the reconstructed output and the target output, thereby learning the inherent semantic association patterns in the data.

[0051] Subsequently, the system calculates its bidirectional consistency constraint loss. The bidirectional consistency constraint loss is an indicator that measures the degree to which two or more semantic representations can be reconstructed from each other. In practice, two related parts of the text data can be selected, such as two texts in different descriptions of the same marketing campaign, or two pieces of feedback from the same customer at different times, generating semantic representations A and B respectively. The reconstruction network first attempts to reconstruct B from A, calculating the loss L1 between the reconstructed B and the original B; then, it attempts to reconstruct A from B, calculating the loss L2 between the reconstructed A and the original A. The bidirectional consistency constraint loss is a combination of L1 and L2, such as summation or a weighted average. The larger the loss value, the higher the inconsistency between the two semantic representations, meaning that the text content they represent has greater semantic contradictions or differences. Finally, by analyzing the calculation results of the bidirectional consistency constraint loss, the system can identify semantic inconsistencies. When the calculated loss value exceeds a preset threshold, a semantic inconsistency problem is determined. This threshold can be dynamically adjusted through historical data analysis, expert experience, or machine learning methods. The identified inconsistencies may include, but are not limited to: conflicting attribute descriptions of the same entity in different texts, contradictions in the logical order of events, and sudden changes in sentiment. These are all deep semantic problems that are difficult to discover using traditional rule-based or simple pattern matching.

[0052] Through the aforementioned technical solution, this application introduces a semantic inconsistency detection unit. This unit can perform refined hierarchical semantic representation of unstructured text data and calculate its bidirectional consistency constraint loss using a trained reconstruction network. This mechanism enables the deep quality analysis model to deeply capture and identify semantic inconsistencies that are difficult to detect using traditional methods, such as contradictions in different text descriptions of the same marketing object or potential logical conflicts in marketing copy. This significantly improves the depth and accuracy of marketing data quality detection, ensuring that marketing decisions are based on more reliable and consistent textual information, thereby avoiding marketing strategy errors caused by semantic biases and improving the effectiveness of marketing activities. Furthermore, through accurate identification of semantic inconsistencies, the system can provide more targeted correction suggestions for the subsequent intelligent data cleaning module, further optimizing the efficiency and effectiveness of the entire data processing flow.

[0053] In some embodiments of this application, the deep quality analysis model further includes an emotional conflict detection unit. This emotional conflict detection unit is used to extract emotional features from text, speech, and image modal data associated with the same data object, map them to a unified emotional space through cross-modal alignment, and calculate the degree of difference to determine emotional conflict-related quality problems.

[0054] Specifically, the sentiment conflict detection unit is a key component of the deep quality analysis model. Its core function is to identify whether there are contradictions or inconsistencies in the emotions expressed by the same data object in different modalities (such as text, voice, and images). This unit provides deeper insights for marketing data quality assessment through multimodal sentiment analysis technology.

[0055] To achieve the above functions, the emotional conflict detection unit first needs to extract emotional features from text, speech, and image modal data associated with the same data object. For text data, deep learning-based natural language processing (NLP) models, such as pre-trained BERT, RoBERTa, or their variants, can be used, fine-tuned with an emotional lexicon and domain knowledge to identify the emotional tendency (e.g., positive, negative, neutral) or specific emotional category (e.g., joy, anger, sadness) in the text. For speech data, speech emotion recognition (SER) technology can be used to extract emotional information from speech by analyzing the prosodic features (e.g., pitch, speech rate, volume), spectral features (e.g., Mel-frequency cepstral coefficients, MFCCs), and acoustic features of the speech signal, combined with models such as recurrent neural networks (RNNs) or convolutional neural networks (CNNs). For image data, computer vision techniques, such as facial expression recognition models, body language analysis models, or scene emotion recognition models based on convolutional neural networks (CNNs), can be used to extract visual emotional features from images. During the extraction process, it is ensured that data from different modalities can be accurately associated with the same data object, for example, by matching timestamps, user IDs, or session IDs.

[0056] After extracting the sentiment features from each modality, the sentiment conflict detection unit needs to map them to a unified sentiment space through cross-modal alignment and calculate the dissimilarity. Cross-modal alignment aims to project the sentiment features extracted from different modalities (text, speech, image) to a common, semantically consistent low-dimensional vector space, i.e., the unified sentiment space. This can be achieved through various techniques. For example, a shared encoder architecture can be used, where input data from different modalities are processed by their respective modality-specific encoders and then mapped to the unified sentiment representation space through a shared fusion layer. Another approach is to utilize contrastive learning or metric learning to train the model so that the sentiment representations of the same data object in different modalities are as close as possible in the unified sentiment space, while the representations of different emotions or different objects are as far apart as possible. Once the sentiment features are mapped to the unified sentiment space, the dissimilarity between them can be calculated. The dissimilarity can be quantified using various distance metrics, such as Euclidean distance, cosine similarity, or Mahalanobis distance. The larger the dissimilarity value, the more significant the sentiment expression conflict between different modalities.

[0057] Ultimately, the emotional conflict detection unit determines emotional conflict quality issues based on the calculated difference. This typically involves setting a predefined threshold. When the calculated difference exceeds this threshold, the system identifies and marks the presence of an emotional conflict quality issue. For example, if the text content expresses high satisfaction, but the associated voice data shows significant negative emotions (such as anger or frustration), and its difference in the unified emotional space exceeds a preset threshold, it will be identified as an emotional conflict.

[0058] Through the aforementioned technical solutions, the deep quality analysis model can transcend single-modal semantic analysis, comprehensively capturing cross-modal sentiment expression conflicts in marketing data. This enables the system to identify more subtle data quality issues that significantly impact marketing decisions, such as inconsistent customer sentiment across different channels or expressions. By accurately identifying sentiment conflict-related quality issues, this application provides more refined and accurate data quality assessment results, thereby helping enterprises to more comprehensively understand customers' true intentions and emotions, optimize marketing strategies, improve customer experience, and effectively mitigate risks arising from misjudgments of sentiment data.

[0059] In some embodiments of this application, the deep quality analysis model further includes a marketing intent feature extraction and association unit, which is used to extract marketing intent features from the data through a multimodal fusion network with denoising bottlenecks, and associate quality issues with semantic inconsistencies and emotional conflicts with specific marketing intent features based on preset mapping rules; wherein, the semantic inconsistency detection unit, the emotional conflict detection unit and the marketing intent feature extraction and association unit share the underlying feature encoder, and are collaboratively trained through multi-task joint loss to simultaneously complete the identification of quality issues and the association of intent features.

[0060] Specifically, the Marketing Intent Feature Extraction and Association Unit aims to identify and extract potential marketing objectives or user intentions from multimodal marketing data. For example, it identifies specific marketing intentions such as "purchase intention," "product inquiry," and "complaint feedback" within user data. This is achieved through a multimodal fusion network with a denoising bottleneck. This network integrates information from different modalities such as text, images, and speech. Through a denoising bottleneck mechanism, it forces the model to learn more essential and robust marketing intent features from the data, effectively filtering out noise and redundant information, thereby improving the accuracy and anti-interference capability of intent recognition. After extracting marketing intent features, this unit, based on preset mapping rules, associates semantic inconsistencies identified by the semantic inconsistency detection unit and emotional conflict issues identified by the emotional conflict detection unit with specific marketing intent features. These mapping rules can be based on domain expert knowledge, historical marketing data analysis results, or obtained through supervised learning model training. For example, a semantic inconsistency issue such as "product description does not match the image" is marked as strongly associated with "product promotion intent."

[0061] Simultaneously, the semantic inconsistency detection unit, the sentiment conflict detection unit, and the marketing intent feature extraction and association unit share a common underlying feature encoder. This means that when processing raw multimodal data, these three functional units will jointly use a basic neural network module to extract general feature representations of the data. This sharing mechanism can effectively reduce the number of model parameters, reduce computational resource consumption, and promote knowledge transfer and sharing between different tasks, enabling the model to learn more comprehensive and generalized feature representations, thereby improving the overall feature extraction efficiency and quality. To optimize the performance of the above three units and ensure their synergy, this application adopts multi-task joint loss for collaborative training. During training, the system calculates the loss of the semantic inconsistency detection task, the loss of the sentiment conflict detection task, and the loss of the marketing intent feature extraction and association task, and then weights and sums these losses to form a total joint loss function. By minimizing this joint loss function, the model can simultaneously optimize all related tasks, allowing each unit to promote each other during the learning process and jointly improve performance. This collaborative training method helps the model better understand the complex relationships in the data, ensuring the synchronicity and consistency of quality problem identification and marketing intent association. Through the collaborative training mechanism of the shared underlying feature encoder and multi-task joint loss, the deep quality analysis model can simultaneously identify quality issues such as semantic inconsistencies and sentiment conflicts in marketing data within a unified framework, and correlate them with specific marketing intent features of the data. This means that upon discovering data quality issues, the system can immediately understand which marketing objectives or user intentions these issues may affect, thus providing more insightful information for subsequent data cleaning, correction, and marketing decisions.

[0062] Through the aforementioned technical solution, this application further enhances the understanding of data value and impact beyond simply identifying marketing data quality issues. Specifically, by introducing a marketing intent feature extraction and association unit, the system goes beyond merely identifying superficial quality problems such as semantic inconsistencies or sentiment conflicts. It delves deeper into the intrinsic connections between these issues and specific marketing objectives. For example, when a semantic inconsistency is detected in a product description, the system can simultaneously identify what might affect a user's "purchase intent" or "product perception intent," making subsequent cleaning and correction more targeted and prioritizing quality issues with the greatest impact on core marketing intent. Furthermore, by sharing the underlying feature encoder with the semantic inconsistency detection unit, sentiment conflict detection unit, and marketing intent feature extraction and association unit, and employing multi-task joint loss for collaborative training, the system not only effectively reduces model complexity and computational costs but, more importantly, enables mutual learning and reinforcement among the tasks. This ensures that while identifying quality issues, the extracted marketing intent features are more accurate and robust; conversely, a deeper understanding of marketing intent can also help to more accurately locate and interpret quality issues. This collaborative mechanism makes quality issue identification and intent association a seamlessly integrated process, greatly enhancing the depth and practicality of marketing data quality analysis. It provides a more refined and guiding basis for subsequent intelligent cleaning, standardization, and marketing strategy formulation, ultimately helping to improve the overall efficiency and effectiveness of marketing activities.

[0063] See Figure 4 In some embodiments of this application, the intelligent data cleaning module includes a rule-driven cleaning engine, an artificial intelligence-driven cleaning engine, a cleaning strategy optimizer, a privacy compliance desensitization unit, and an edge lightweight cleaning acceleration unit.

[0064] The rule-driven data cleaning engine performs standardized data cleaning operations based on a general rule layer and a marketing business rule layer. This engine identifies and corrects formatting errors, value range errors, and logical errors in the data through preset, configurable rule sets. The general rule layer can include basic rules such as data type validation, format standardization, and null value handling; the marketing business rule layer can formulate more refined business rules for specific marketing scenarios, such as customer ID format standardization, product name standardization, and campaign code validation. Rule-driven operation ensures the determinism and controllability of the data cleaning process, making it suitable for handling known and clearly defined data quality issues.

[0065] An AI-driven data cleaning engine is used to perform missing value imputation, outlier correction, duplicate record matching and merging, and semantic-level deep cleaning. This engine utilizes AI technologies such as machine learning and deep learning to handle complex, ambiguous, or unstructured data quality problems that traditional rules struggle to address effectively. For example, for missing values, predictive imputation based on data patterns and relevant features can be performed instead of simple deletion or replacement with fixed values; for outliers, statistical models, clustering algorithms, or classification models can be used for identification and intelligent correction suggestions; for duplicate records, fuzzy matching algorithms can be used to identify and merge near-duplicate items; and for semantic-level deep cleaning, natural language processing techniques can be combined to perform entity linking, relationship verification, and semantic error correction on text data.

[0066] The data cleaning strategy optimizer iteratively optimizes the AI ​​model and data cleaning rule base based on data cleaning feedback. This optimizer analyzes and learns from the collected data quality assessment results and user feedback to refine the current data cleaning strategy. For example, it can employ reinforcement learning or meta-learning methods to automatically adjust rule priorities and weights, or optimize AI model parameters and select more suitable algorithms based on cleaning performance (such as precision, recall, and F1 score). Through this feedback loop mechanism, the data cleaning strategy can continuously and adaptively improve, thereby enhancing the accuracy and efficiency of data cleaning.

[0067] The privacy compliance de-identification unit is used in conjunction with the data cleansing process to perform de-identification of marketing data. During or after data cleansing, this unit de-identifies marketing data containing personally identifiable information or other sensitive information to ensure that the data complies with relevant privacy regulations (such as GDPR and CCPA) and internal corporate compliance requirements in subsequent use and analysis. De-identification techniques may include data anonymization, pseudonymization, generalization, suppression, or encryption to ensure that user privacy is protected without affecting data usability.

[0068] The edge lightweight cleaning acceleration unit performs real-time lightweight cleaning computations at the edge. This unit offloads some cleaning tasks to the edge devices that receive the data source or data acquisition, such as performing preliminary data format validation, simple deduplication, and data type conversion. By preprocessing at the edge, the amount of data that needs to be transmitted to the central processing system can be significantly reduced, lowering network bandwidth and the load on the central server, while achieving real-time or near real-time cleaning, making it particularly suitable for marketing data scenarios with high timeliness requirements.

[0069] Through the aforementioned technical solutions, this system, via the collaborative operation of a rule-driven cleaning engine and an AI-driven cleaning engine, can comprehensively and flexibly address various structured and unstructured quality issues in marketing data, significantly improving the accuracy and coverage of cleaning. The feedback loop mechanism introduced by the cleaning strategy optimizer enables the cleaning strategy to continuously learn and adaptively optimize, ensuring continuous improvement in cleaning effectiveness. The close integration of the privacy compliance de-identification unit with the cleaning process technically guarantees the privacy and compliance of marketing data, effectively reducing the risk of data leakage. The edge-lightweight cleaning acceleration unit moves some cleaning tasks to the edge, greatly improving cleaning efficiency and reducing the pressure on central processing, especially suitable for marketing scenarios requiring real-time response. Overall, this intelligent data cleaning module, through a multi-dimensional and collaborative cleaning mechanism, significantly improves the comprehensiveness, accuracy, efficiency, and compliance of marketing data cleaning, providing a high-quality data foundation for subsequent marketing analysis and decision-making.

[0070] See Figure 5 In some embodiments of this application, the AI-driven cleaning engine includes a missing value intelligent filling unit, an outlier identification and correction unit, a duplicate record fuzzy matching and merging unit, a semantic-level deep cleaning unit, and a marketing intent association correction unit. The semantic-level deep cleaning unit combines a pre-trained language model with a domain knowledge graph to complete entity linking, relationship verification, and semantic error correction; the marketing intent association correction unit is used to perform targeted correction of quality issues based on extracted marketing intent features.

[0071] Specifically, the intelligent missing value imputation unit aims to address the prevalent problem of missing values ​​in marketing data. It works by intelligently inferring and filling in missing data points by analyzing patterns and contextual information in existing data. For example, statistical methods (such as mean, median, and mode imputation) can be used, or machine learning models (such as regression models, classification models, and generative adversarial networks (GANs)) can be employed to predict missing values ​​based on other relevant features. More advanced implementations can consider the temporal or relational nature of the data; for instance, for customer behavior data, imputation can be based on historical user behavior or similar user behavior to ensure the reasonableness and accuracy of the imputed values.

[0072] The outlier identification and correction unit is used to detect and handle outliers or irregularities in marketing data. Outliers may be caused by data entry errors, system malfunctions, or special events, and can severely affect the accuracy of data analysis. Identification methods can include statistical methods (such as Z-score, IQR), distance-based methods (such as K-nearest neighbors), density-based methods (such as LOF), and model-based methods (such as Isolation Forest, One-Class SVM). Correction strategies can include replacing outliers with reasonable values ​​(such as the mean, median, or nearest neighbors), truncating outliers, and in some cases, marking and isolating outliers for further analysis.

[0073] The Fuzzy Matching and Merging Unit for Duplicate Records aims to identify and integrate duplicate records in marketing data caused by different sources and entry methods. Due to inconsistencies in data entry, duplicate records are often not completely identical but have subtle differences (i.e., fuzzy duplication). This unit evaluates the similarity between records using various similarity calculation methods (such as string edit distance, Jaccard similarity, cosine similarity, etc.) and combines them with machine learning models for fuzzy matching. Once duplicate records are identified, the system merges them into a single record according to a preset merging strategy (such as retaining the latest record, merging all non-conflicting fields, manual review, etc.), thereby improving data consistency and uniqueness.

[0074] Semantic deep cleaning is crucial for addressing deeper semantic issues in marketing data. It combines advanced pre-trained language models with domain knowledge graphs. Pre-trained language models (such as BERT and GPT series) understand the context and semantic information of text, identifying grammatical errors, semantic ambiguities, or expressions that defy common sense. Domain knowledge graphs provide structured marketing domain knowledge, including entities (such as products, customers, and activities) and their relationships. Entity linking maps entities in the text to standard entities in the knowledge graph, resolving inconsistencies in entity referencing. Relationship verification validates the correctness of relationships between entities, for example, checking if a product belongs to a particular brand. Semantic correction corrects inaccurate or unreasonable semantic expressions based on the understanding of the language model and knowledge graph, ensuring the semantic accuracy of marketing data.

[0075] The Marketing Intent Association Correction Unit is a key function of this system optimized for the characteristics of marketing data. Based on marketing intent features extracted from the data, it performs targeted corrections on identified quality issues. For example, if a customer's address information contains a minor error, but the associated marketing intent features indicate that the customer has a strong interest in a promotional activity in a certain area, the correction will prioritize ensuring the accuracy of the address information to guarantee the precise delivery of promotional information. This unit ensures that the data cleaning process not only focuses on the surface accuracy of the data, but more importantly, that the cleaned data better serves specific marketing goals and strategies, avoiding damage to the marketing value of the data due to indiscriminate correction.

[0076] Through the aforementioned technical solutions, the AI-driven data cleaning engine can not only handle basic issues such as missing values, outliers, and duplicate records, but more importantly, it can delve into the semantic level of marketing data for cleaning and correction. Specifically, the semantic-level deep cleaning unit, by combining pre-trained language models and domain knowledge graphs, can accurately complete entity linking, relationship verification, and semantic error correction, effectively solving deep-seated semantic errors and inaccurate entity relationships that traditional cleaning methods struggle to address, significantly improving the semantic accuracy and consistency of marketing data. Furthermore, the introduction of the marketing intent association correction unit allows the cleaning process to target quality issues based on extracted marketing intent features, ensuring that the cleaned data is not only technically accurate but also strategically aligned with marketing objectives, avoiding potential marketing value loss due to generic cleaning rules. This makes the cleaned marketing data more insightful, more effectively supporting advanced marketing applications such as personalized recommendations, precision marketing, and customer relationship management, thereby significantly improving the efficiency and effectiveness of marketing campaigns.

[0077] In some embodiments of this application, the duplicate record fuzzy matching and merging unit is used to identify approximately duplicate items using an NLP similarity algorithm and perform a merging operation.

[0078] Specifically, NLP similarity algorithms are techniques used to measure the degree of similarity between textual data. Their implementation can include, but is not limited to: calculating cosine similarity based on the Bag-of-Words model combined with TF-IDF (Term Frequency-Inverse Document Frequency) weights; using word embeddings techniques (such as Word2Vec and GloVe) to map words to a high-dimensional vector space and then calculating the cosine similarity or Euclidean distance between vectors; or using more advanced pre-trained language models (such as BERT and GPT series) to extract semantic representations of the text and then calculate their similarity. Furthermore, for approximate matching at the string level, algorithms such as edit distance (such as Levenshtein distance) and Jaccard similarity can also be used. These algorithms can effectively handle common spelling variations, abbreviations, synonyms, or different expressions in marketing data, thereby identifying semantically or content-wise highly related near-repeating items.

[0079] Identifying near-duplicate items involves using NLP similarity algorithms to perform pairwise comparisons or clustering analysis on records in marketing data. When the similarity between two or more records reaches a preset threshold, they are identified as near-duplicate items. For example, the customer names "Zhang Sanfeng" and "Zhang San," or the product descriptions "smartphone" and "intelligent mobile phone," can be calculated using NLP similarity algorithms to determine their high similarity, thus identifying them as near-duplicate records pointing to the same entity or concept.

[0080] Performing a merge operation refers to integrating identified duplicate records into a single, high-quality record. Merging strategies can include: retaining the most recent or complete record; integrating non-conflicting information from all duplicate records; or selectively merging based on pre-defined business rules, such as retaining the most detailed address information for customer addresses and accumulating purchase history. This process aims to eliminate redundancy and ensure data uniqueness and consistency.

[0081] Through the aforementioned technical solution, the fuzzy matching and merging unit for duplicate records leverages the powerful capabilities of NLP similarity algorithms to overcome the limitations of traditional exact matching, effectively identifying near-duplicate items in marketing data caused by spelling errors, differences in expression, or semantic similarities. This significantly improves the accuracy and coverage of data cleaning, avoiding market analysis biases and wasted marketing resources due to data redundancy and inconsistency. Intelligent identification and merging of near-duplicate items ensures the uniqueness and consistency of marketing data, providing a more reliable and high-quality data foundation for subsequent marketing strategy formulation, customer profiling analysis, and personalized recommendations, thereby improving the efficiency and effectiveness of the entire intelligent data cleaning module.

[0082] In some embodiments of this application, the standardization processing and mapping module includes a marketing data standard library, an intelligent mapping engine, and a federated cleaning and alignment unit. The marketing data standard library, as a core component of the system, aims to centrally define and manage unified enterprise-level marketing data standards. This standard library can be a structured database, a configuration management system, or a knowledge graph-based store, which details various specifications for marketing data, such as data types, format requirements, encoding rules, units of measurement, business terminology definitions, customer classification systems, and product classification systems. By establishing and maintaining such a unified standard library, it is possible to ensure that all accessed and processed marketing data maintains semantic and structural consistency, providing a solid foundation for subsequent data cleaning, mapping, and analysis.

[0083] Building upon this foundation, this application also proposes an intelligent mapping engine, whose main function is to automatically map and transform source data fields into standard fields. This engine accomplishes this task by comprehensively utilizing metadata and semantic analysis technologies. Specifically, the metadata-driven mechanism allows the engine to acquire structural information, data types, and descriptions of source data fields, while semantic analysis, through natural language processing (NLP) techniques, ontology matching, or machine learning models, understands the business meaning of the source data fields and matches them with standard fields defined in the marketing data standard library. For example, the engine can learn historical mapping rules or recommend or automatically execute mappings by calculating the similarity between field names and descriptive text. This intelligent mapping method significantly reduces manual intervention, improves the accuracy and efficiency of mapping, and is particularly suitable for handling large-scale, highly complex heterogeneous data sources.

[0084] Furthermore, this application introduces a federated cleansing and alignment unit, which innovatively completes the cleansing and standard alignment of cross-platform marketing data under a federated learning model. In many scenarios, marketing data may be scattered across different platforms, departments, or partners. Due to privacy protection and data security restrictions, this data cannot be centrally processed in one place. The federated cleansing and alignment unit, through the federated learning paradigm, allows each data owner to clean and standardize their data locally, sharing only model parameters or aggregated intermediate results with the central coordinator or participants, rather than the raw data. In this way, without exposing sensitive marketing data, all parties can collaboratively train a global cleansing and alignment model, ensuring that marketing data on different platforms can ultimately be aligned to a unified standard, thereby achieving effective integration and utilization of cross-platform data while strictly adhering to data privacy and compliance requirements.

[0085] Through the aforementioned technical solutions, this system effectively addresses the challenges of efficiency, accuracy, privacy compliance, and cross-platform collaboration in data standardization within a multi-source, heterogeneous marketing data environment. The introduction of a marketing data standard library provides the entire system with clear and unified data specifications, fundamentally ensuring data consistency. The intelligent mapping engine significantly improves the efficiency and accuracy of converting source data to standard data through automation and intelligence, reducing human error and workload. More importantly, the federated cleaning and alignment unit, while ensuring data privacy and security, achieves collaborative cleaning and standard alignment of cross-platform marketing data, enabling the effective integration and utilization of data scattered across different entities, thereby forming a more comprehensive and accurate marketing data view. Overall, the collaborative work of these components ensures high-quality and high-availability marketing data throughout its lifecycle, providing a solid data foundation for precision marketing, customer insights, and business decision-making, especially suitable for complex scenarios requiring the processing of large amounts of sensitive and distributed marketing data.

[0086] In some embodiments of this application, the marketing data standard library covers data formats, encoding rules, units of measurement, business terminology, customer classification systems, and product classification system standards.

[0087] Specifically, data format refers to the standardized specification of data type, length, precision, and storage method for various fields in marketing data. For example, date fields should be uniformly formatted as "YYYY-MM-DD", numeric fields should specify the number of decimal places, and text fields should specify the maximum length. This ensures structural consistency of data from different sources, facilitating subsequent processing and storage. Encoding rules involve the standardized specification of character sets and encoding methods for specific business codes (such as product SKUs, customer IDs, and regional codes) in the data. For example, all text data uniformly uses UTF-8 encoding, and product SKU encoding follows the specific rule of "brand-series-model" to ensure correct parsing and identification of data during transmission and processing. Measurement units specify the unified unit of measurement and conversion relationships for numerical data such as quantity, amount, and time in marketing data. For example, all amounts are uniformly expressed in "yuan", and time is uniformly expressed in "seconds" or "days," ensuring that values ​​from different data sources can be directly compared and aggregated, avoiding calculation errors caused by inconsistent units. The business terminology system establishes a unified marketing business glossary within the enterprise, clearly defining and standardizing the use of core business concepts such as "customer," "order," "conversion rate," and "event." This helps eliminate discrepancies in understanding the same business concepts across different departments or systems, improving the accuracy and readability of data analysis reports. The customer classification system establishes unified standards for stratifying, classifying, or tagging customers. For example, categories such as "new customers," "active customers," "high-value customers," and "churned customers" can be defined based on dimensions like customer spending behavior, lifecycle stage, and value contribution, with clear classification rules, providing a unified customer profile foundation for precision marketing. The product classification system standard specifies unified guidelines for classifying, grading, and defining the attributes of the enterprise's products. For example, a multi-level product classification tree (such as "electronic products - mobile phones - smartphones") can be established, and common product attributes (such as color, size, and material) can be defined, ensuring consistent management and display of product data across different systems.

[0088] Through the aforementioned technical solutions, the marketing data standard library is no longer an abstract "unified standard," but rather a comprehensive and specific specification encompassing data formats, encoding rules, units of measurement, business terminology, customer classification systems, and product classification systems. This significantly enhances the operability and coverage of the data standards, ensuring that multi-source, heterogeneous marketing data can be accurately understood and processed during the access, cleaning, and mapping processes. Specifically, clear data formats and encoding rules prevent data parsing errors; unified units of measurement and business terminology eliminate semantic ambiguity, guaranteeing the accuracy of data calculation and analysis; and standardized customer and product classification systems provide a unified and reliable data foundation for the formulation and execution of marketing strategies. Ultimately, this enables the entire intelligent cleaning and standardization processing system to process complex marketing data more efficiently and accurately, significantly improving data quality and the value of subsequent data applications.

[0089] In some embodiments of this application, the quantitative data quality assessment and feedback module includes a multi-dimensional assessment indicator system, a full-link quality scoring unit, and a quality feedback closed-loop unit. The multi-dimensional assessment indicator system includes indicators for completeness, accuracy, consistency, timeliness, effectiveness, and uniqueness. The full-link quality scoring unit is used to perform quality scoring and generate quality reports at the data access and data cleaning nodes. The quality feedback closed-loop unit is used to feed the assessment results back to the data detection, cleaning, and standard setting stages to achieve iterative quality optimization.

[0090] Specifically, the multi-dimensional evaluation indicator system aims to provide a comprehensive and detailed framework for measuring data quality. This system decomposes data quality into several key dimensions. For example, the completeness indicator measures the fillability of key fields in data records, which can be assessed by calculating the proportion of non-empty fields or comparing it with business rule requirements; the accuracy indicator measures the degree to which data values ​​conform to real-world conditions, which can be verified by comparing with authoritative data sources or through preset validation rules; the consistency indicator measures the logical uniformity of data across different systems, different points in time, or different related data, such as checking whether the representation of the same customer information is consistent in different business systems; the timeliness indicator measures the time delay from data generation to availability, ensuring that data meets business requirements for real-time performance; the validity indicator measures whether data meets its intended business purpose and format requirements, such as checking whether a phone number is in a valid format; and the uniqueness indicator identifies and quantifies the number of duplicate records in the dataset, ensuring that each entity is represented only once. Through these detailed indicators, the system can conduct multi-faceted and in-depth analysis of data quality.

[0091] The end-to-end quality scoring unit is responsible for quality assessment and scoring at key nodes in the data processing flow. For example, after the data access and adaptation module completes data transformation, this unit immediately performs a preliminary quality assessment of the accessed data, generating a first-stage quality score and report. Subsequently, after the intelligent data cleaning module completes the cleaning operation, this unit performs another quality assessment of the cleaned data, generating a second-stage quality score and report. This phased quality scoring mechanism allows the system to clearly track changes in data quality throughout the entire processing chain and identify at which stage quality issues are introduced or improved. The generated quality report presents the scores of various indicators, trend analysis, and specific problems found in a structured format, providing data support for subsequent optimization.

[0092] The quality feedback closed-loop unit is the core mechanism for continuous data quality improvement. This unit receives and analyzes quality reports generated by the end-to-end quality scoring unit, identifying substandard quality dimensions or persistent quality issues. Based on these analyses, the quality feedback closed-loop unit feeds back specific optimization suggestions or adjustment instructions to the data probing and monitoring module, the intelligent data cleaning module, and the standardization and mapping module. For example, if accuracy metrics remain consistently low, the feedback closed-loop unit might suggest adjusting the threshold of the dynamic rule engine in the data probing module or updating the model of the machine learning probing unit; if integrity issues are prominent, it might trigger the missing value intelligent imputation unit in the intelligent data cleaning module to process them more aggressively; if consistency issues are related to data standards, they will be fed back to the standardization and mapping module to revise the mapping rules of the marketing data standard library or the intelligent mapping engine. Through this mechanism, the system can continuously adjust and optimize its probing, cleaning, and standardization strategies based on actual data quality performance, thereby achieving continuous iterative optimization of data quality.

[0093] Through the aforementioned technical solution, this application introduces a multi-dimensional evaluation index system, enabling comprehensive and detailed quantitative evaluation of multiple key dimensions of marketing data, such as completeness, accuracy, consistency, timeliness, effectiveness, and uniqueness. This overcomes the limitations of single-indicator evaluation, allowing for more accurate identification and location of data quality issues. The end-to-end quality scoring unit performs quality scoring and generates quality reports at the data access and data cleaning nodes, providing a clear view of data quality at different processing stages and facilitating timely discovery and location of problem sources. More importantly, the quality feedback closed-loop unit effectively feeds back the evaluation results to the data detection, cleaning, and standard setting stages, forming a continuous quality iteration and optimization mechanism. This not only ensures continuous improvement in data quality but also enables the system to adaptively adjust detection rules, cleaning strategies, and standard definitions based on feedback data, thereby significantly improving the overall system's intelligence level in processing marketing data and the value of data assets.

[0094] The second aspect of this application proposes a method for intelligent cleaning and standardization of multi-source heterogeneous marketing data. See also... Figure 6 The method includes the following steps: Unified access and adaptation of multi-source heterogeneous marketing data; conversion of multi-source heterogeneous marketing data into unified intermediate representation layer data; and preliminary extraction and structuring of unstructured marketing data. The marketing data after unified access and adaptation is subjected to data quality detection and monitoring, automatically identifying data quality problems, extracting marketing intent features, and performing real-time monitoring and multi-level early warning; It adopts a dual-engine architecture that integrates rule-driven and AI-driven approaches to perform intelligent data cleaning on the quality issues identified by quality detection, while simultaneously performing marketing data privacy compliance desensitization and real-time edge cleaning acceleration. Based on the preset unified marketing data standard, the marketing data after intelligent cleaning is subjected to standardized processing and field mapping transformation, and cross-platform data cleaning and alignment is completed in federated learning mode. The standardized marketing data is subjected to multi-dimensional quantitative quality assessment, and the assessment results are fed back to the data quality detection, intelligent data cleaning and standardization processes to form a closed loop for quality optimization.

[0095] This method addresses the problems of low data quality, insufficient standardization, fragmented processing, difficulty in identifying unstructured data quality risks, and lack of continuous feedback optimization loops in existing technologies for processing multi-source heterogeneous marketing data. It innovatively constructs a fully automated processing mechanism to solve these problems. Specifically, in the unified access and adaptation phase, multi-source heterogeneous marketing data is converted into unified intermediate representation layer data. Simultaneously, unstructured marketing data undergoes preliminary extraction and structuring, effectively eliminating the fragmented processing caused by data source heterogeneity. In the data quality detection and monitoring phase, marketing intent features are extracted while automatically identifying data quality issues. This enables deep identification of semantic inconsistencies and emotional conflicts in unstructured data such as text and images, overcoming the limitations of traditional solutions that lack effective technical means. The intelligent data cleaning phase employs a dual-engine architecture that integrates rule-driven and AI-driven approaches. This not only adaptively handles quality issues such as missing value filling and outlier correction but also simultaneously performs privacy-compliant de-identification processing and real-time edge-side cleaning acceleration, significantly improving cleaning efficiency and data security. The standardization and mapping phase, based on a pre-defined unified marketing data standard, completes cross-platform data cleaning and alignment under a federated learning model, ensuring data consistency and interoperability across different business systems. Finally, multi-dimensional quantitative quality assessment is used, and the results are fed back to the front-end to form an optimization loop, enabling data quality to continuously and automatically improve along with business development. The core innovation of this embodiment lies in combining a rule-driven and AI-driven dual-engine architecture with marketing intent feature extraction in a fully automated manner, and introducing cross-platform data cleaning and alignment under a federated learning model. This effectively solves key problems in multi-source heterogeneous marketing data processing, such as process gaps, difficulty in identifying unstructured data quality risks, and the lack of a continuous feedback optimization loop, achieving the core effect of continuous and automatic data quality improvement along with business development.

[0096] The following is a more specific example to illustrate the technical solution proposed in this application in more detail: Suppose a large retail company needs to process its marketing data from multiple channels to improve the accuracy of personalized marketing campaigns. These data sources include: structured customer profiles and purchase histories from a customer relationship management (CRM) system, structured transaction records and browsing behavior from e-commerce platforms, unstructured text comments and images from social media platforms, and unstructured chat logs from a customer service center. The main problem the company faces is that this data comes in various formats, is of inconsistent quality, and lacks a unified standard, making it difficult to form a complete customer view and effectively utilize the data.

[0097] First, the data access and adaptation module is responsible for the unified access of these multi-source heterogeneous data. This module develops dedicated adapters for CRM databases, e-commerce APIs, social media interfaces, and customer service system log files through a multimodal adapter framework. For example, one adapter extracts relational data from the CRM system, while another adapter retrieves JSON-formatted text and image data from the social media API. The metadata-driven management unit centrally manages the metadata of these data sources, such as the definition of the "customer name" field in the CRM and the format of "post time" in social media data. The unified data interface layer provides standardized data access interfaces, converting all data into a unified intermediate representation layer. For product tags in social media images or customer feedback reports in PDF format, the OCR parsing unit in the unstructured data parsing engine performs text recognition. For social media comments and customer service chat logs, the NLP parsing unit extracts key entities (such as product names and brands), relationships (such as "interested in a product"), and preliminary structured information (such as sentiment). Compared to traditional solutions that require manually writing numerous scripts to handle different data sources and formats, and where unstructured data is often simply stored and cannot be effectively utilized, this system achieves automated and efficient data access and preliminary structuring through a unified interface and dedicated adapters.

[0098] Next, the data quality detection and monitoring module identifies and monitors quality issues in the incoming data in real time. The dynamic rule engine executes preset business rules, such as requiring the "customer age" field to be between 18 and 99, and can adaptively adjust the threshold. The machine learning detection unit uses unsupervised learning algorithms to identify anomalous purchase amounts (outliers) in CRM data or unusual purchase sequences (logical errors) in e-commerce transaction records. The deep quality analysis model focuses on unstructured data: the semantic inconsistency detection unit performs hierarchical semantic representation of text in customer service chat logs. For example, in the same conversation, if a customer first expresses "very satisfied with product A" and then complains "product A is of poor quality," the system calculates the bidirectional consistency constraint loss through a trained reconstruction network to identify the semantic contradiction. The sentiment conflict detection unit associates the same customer's social media comments (text), product review images (images), and voice customer service records (voices), extracts sentiment features, maps them to a unified sentiment space through cross-modal alignment, and calculates the degree of difference to determine sentiment conflict-related quality issues. The marketing intent feature extraction and association unit extracts marketing intent features such as "purchase intention" and "complaint intention" through a multimodal fusion network, and associates quality issues such as semantic inconsistency or sentiment conflict with specific marketing intent features. For example, semantic inconsistency under the intent of "product consultation" is marked as a high-priority issue. The visualization monitoring and early warning platform displays a data quality dashboard in real time, for example, showing "customer name completeness" at 95% and "product description consistency" at 80%. When any indicator falls below a preset threshold, the system immediately triggers multi-level warnings. Existing solutions typically can only detect simple errors in structured data through hard-coded rules, and are helpless against semantic and sentiment problems in unstructured data, lacking a real-time, intelligent early warning mechanism. This system can deeply understand the content of unstructured data and perform multi-dimensional, intelligent quality detection.

[0099] Subsequently, the intelligent data cleaning module adaptively cleans the detected quality issues. The rule-driven cleaning engine performs standardized cleaning on CRM and e-commerce data based on a general rule layer (e.g., removing special characters, standardizing date formats) and a marketing business rule layer (e.g., standardizing all "phone number" fields to 11 digits). The AI-driven cleaning engine handles more complex cleaning tasks: the missing value intelligent filling unit intelligently fills in the missing "customer occupation" field in the CRM by combining customer purchase history and browsing behavior. The outlier identification and correction unit corrects purchase amounts in e-commerce transaction data that significantly exceed the normal range. The duplicate record fuzzy matching and merging unit uses NLP similarity algorithms to identify and merge nearly duplicate customer records in the CRM, such as "Zhang Sanfeng," "Zhang Sanfeng (Mr.)," and "Zhang Sanfeng-VIP," forming a unified customer profile. The semantic-level deep cleaning unit combines a pre-trained language model and domain knowledge graph to perform entity linking on product names in customer service chat logs (e.g., linking "Apple mobile phone" to "iPhone"), verifying the relationship between customer descriptions and actual product attributes, and correcting semantic errors. The marketing intent association correction unit prioritizes correcting semantic inconsistencies related to complaints based on previously extracted "complaint intention" features. The data cleaning strategy optimizer iteratively optimizes the AI ​​model and cleaning rule base based on the data quality assessment results and user feedback after cleaning. The privacy compliance desensitization unit works in conjunction with the cleaning process to desensitize sensitive customer information (such as phone numbers and ID numbers). The edge lightweight cleaning acceleration unit performs preliminary lightweight cleaning, such as format validation, on real-time sales data generated on edge devices like store POS machines, reducing the amount of data transmitted to the central system and the cleaning pressure. Existing cleaning tools mostly rely on manual rules or simple statistical methods, making it difficult to handle complex semantic problems and unstructured data, and lacking privacy protection and edge processing capabilities. This system, through a dual-engine architecture and AI capabilities, achieves smarter and more comprehensive cleaning while balancing privacy and efficiency.

[0100] Subsequently, the standardization and mapping module, based on a pre-defined unified marketing data standard, performs standardization processing and field mapping transformation on the cleaned data. The marketing data standard library defines enterprise-level unified marketing data standards, such as customer classification systems (VIP, regular customers), product classification systems (electronics, apparel), units of measurement (sales revenue standardized to RMB), and business terminology (standardizing "promotional activity" to "Marketing Campaign"). The intelligent mapping engine, based on metadata and semantic analysis, automatically maps cleaned source data fields to standard fields. For example, it maps "customer name" in CRM to "unified customer name" in the standard library, and "order amount" in e-commerce to "unified sales revenue" in the standard library. The federated cleansing and alignment unit, in federated learning mode, can complete the cleansing and standard alignment of cross-platform marketing data with partners (such as logistics companies) without exchanging raw data, ensuring data consistency across different platforms. Traditional standardization processes are usually manual or semi-automatic, inefficient and error-prone, especially when sharing data across platforms. This system achieves efficient and secure standardization through intelligent mapping and federated learning.

[0101] Finally, the quantitative data quality assessment and feedback module performs multi-dimensional quantitative quality assessments on the standardized data. The multi-dimensional assessment indicator system includes indicators such as completeness (e.g., the fill rate of the customer address field), accuracy (e.g., the matching degree between product price and actual price), consistency (e.g., whether the gender of the same customer is consistent in different systems), timeliness (e.g., data update frequency), validity (e.g., the proportion of invalid customer records), and uniqueness (e.g., the proportion of duplicate customer records). The end-to-end quality scoring unit performs quality scoring at each node, including data access, cleaning, and standardization, and generates detailed quality reports. The quality feedback closed-loop unit feeds the assessment results back to the data detection module (e.g., adjusting detection rules if low accuracy is found in a certain type of data), the cleaning module (e.g., optimizing cleaning strategies if duplicate record merging is ineffective), and the standard setting stage (e.g., revising standards if unclear standard definitions lead to consistency issues), forming a continuous quality iteration and optimization mechanism. Existing solutions often have isolated, one-off quality assessments, lacking comprehensive monitoring and continuous feedback optimization capabilities throughout the entire data lifecycle. This system achieves continuous improvement in data quality through quantitative assessment and closed-loop feedback.

[0102] Through the above examples, the system has achieved full-process automation from multi-source heterogeneous data access, intelligent quality detection, adaptive cleaning, standardized processing to quantitative evaluation and feedback. It effectively solves the problems faced by enterprises when using marketing data, such as low data quality, insufficient standardization, and difficulty in effectively utilizing unstructured data, and significantly improves the credibility and value of data analysis.

[0103] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A multi-source heterogeneous intelligent cleaning and standardization processing system for marketing data, characterized in that, include: The data access and adaptation module is used to convert multi-source heterogeneous data into unified intermediate representation layer data, and to complete the initial extraction and structuring of unstructured data. The data quality detection and monitoring module is used to automatically identify data quality problems and achieve real-time monitoring and early warning. The intelligent data cleaning module is used to adaptively clean data with a dual-engine architecture to address data quality issues, and simultaneously performs privacy compliance de-identification processing on marketing data. The standardization processing and mapping module is used to complete data mapping and transformation based on a unified marketing data standard. The quantitative data quality assessment and feedback module is used to complete the quantitative scoring of data quality and form a feedback loop. The data quality detection and monitoring module includes: A dynamic rules engine is used to execute configurable business rules and achieve adaptive threshold adjustment; The machine learning detection unit is used to identify outliers and isolated points using unsupervised learning algorithms and to discover logical errors using sequence pattern mining. Deep quality analysis models are used to learn semantic features and identify quality problems in unstructured data using deep learning models. A visual monitoring and early warning platform is used to display data quality indicators and implement multi-level early warnings; The deep quality analysis model includes a semantic inconsistency detection unit, which performs hierarchical semantic representation on unstructured text data and calculates its bidirectional consistency constraint loss through a trained reconstruction network to identify semantic inconsistencies.

2. The system according to claim 1, characterized in that, The data access and adaptation module includes a multimodal adapter framework, a metadata-driven management unit, and a unified data interface layer; The multimodal adapter framework is used to develop dedicated adapters for different data sources and data formats, converting data into unified intermediate representation layer data; The metadata-driven management unit is used to centrally manage the metadata information of the access data source; The unified data interface layer is used to provide a standardized data access interface and connect to the unstructured data parsing engine to achieve information extraction and structuring.

3. The system according to claim 2, characterized in that, The unstructured data parsing engine includes an OCR parsing unit and an NLP parsing unit. The unstructured data parsing engine is used to perform preliminary extraction and structuring of text and images.

4. The system according to claim 1, characterized in that, The deep quality analysis model also includes an emotional conflict detection unit, which is used to extract emotional features from text, speech and image modal data that are associated with the same data object, map them to a unified emotional space through cross-modal alignment and calculate the degree of difference to determine emotional conflict-related quality problems.

5. The system according to claim 4, characterized in that, The deep quality analysis model also includes a marketing intent feature extraction and association unit, which is used to extract marketing intent features from the data through a multimodal fusion network with a denoising bottleneck, and associate and label the quality problems of semantic inconsistency and emotional conflict with specific marketing intent features based on preset mapping rules. The semantic inconsistency detection unit, the sentiment conflict detection unit, and the marketing intent feature extraction and association unit share the underlying feature encoder and are trained collaboratively through multi-task joint loss to simultaneously complete the identification of quality problems and the association of intent features.

6. The system according to claim 1, characterized in that, The intelligent data cleaning module includes a rule-driven cleaning engine, an artificial intelligence-driven cleaning engine, a cleaning strategy optimizer, a privacy compliance desensitization unit, and an edge lightweight cleaning acceleration unit. The rule-driven cleaning engine is used to perform standardized cleaning operations based on the general rule layer and the marketing business rule layer; The AI-driven cleaning engine is used to complete missing value filling, outlier correction, duplicate record matching and merging, and semantic-level deep cleaning. The cleaning strategy optimizer is used to iteratively optimize the artificial intelligence model and the cleaning rule base based on cleaning feedback data. The privacy compliance desensitization unit is used to perform marketing data desensitization processing in conjunction with the cleansing process; The edge lightweight cleaning acceleration unit is used to perform real-time lightweight cleaning calculations on the edge side.

7. The system according to claim 6, characterized in that, The AI-driven cleaning engine includes a missing value intelligent filling unit, an outlier identification and correction unit, a duplicate record fuzzy matching and merging unit, a semantic-level deep cleaning unit, and a marketing intent association correction unit. The semantic-level deep cleaning unit is used to combine pre-trained language models and domain knowledge graphs to complete entity linking, relationship verification and semantic error correction. The marketing intent association correction unit is used to perform targeted correction of quality issues based on the extracted marketing intent features; The duplicate record fuzzy matching and merging unit is used to identify approximately duplicate items using an NLP similarity algorithm and perform a merging operation.

8. The system according to claim 1, characterized in that, The standardization processing and mapping module includes a marketing data standard library, an intelligent mapping engine, and a federated cleaning and alignment unit. The aforementioned marketing data standard library is used to define a unified standard for enterprise-level marketing data. The intelligent mapping engine is used to automatically map and convert source data fields to standard fields based on metadata and semantic analysis; The federated cleaning and alignment unit is used to clean and standardize cross-platform marketing data in federated learning mode.

9. The system according to claim 8, characterized in that, The marketing data standard library covers data formats, encoding rules, units of measurement, business terminology, customer classification systems, and product classification system standards.

10. A method for intelligent cleaning and standardization of multi-source heterogeneous marketing data, characterized in that, include: Unified access and adaptation of multi-source heterogeneous marketing data; conversion of multi-source heterogeneous marketing data into unified intermediate representation layer data; and preliminary extraction and structuring of unstructured marketing data. The system performs data quality detection and monitoring on marketing data after unified access and adaptation, automatically identifies data quality problems, extracts marketing intent features, and performs real-time monitoring and multi-level early warning. It adopts a dual-engine architecture that integrates rule-driven and AI-driven approaches to intelligently clean quality issues identified by quality detection, while simultaneously performing marketing data privacy compliance desensitization and real-time edge-side cleaning acceleration. Based on the preset unified standard for marketing data, the marketing data after intelligent cleaning is subjected to standardized processing and field mapping transformation, and cross-platform data cleaning and alignment is completed in federated learning mode. The standardized marketing data is subjected to multi-dimensional quantitative quality assessment, and the assessment results are fed back to the data quality detection, intelligent data cleaning and standardization process to form a closed loop for quality optimization.

Images