An all-channel data middle platform assisted construction method and system

Abstract

The application discloses a full-channel data middle platform auxiliary construction method and system, relates to the technical field of data processing, and accesses multiple-channel data sources, stores collected metadata into an original data pool after converting the metadata into an intermediate data vector, intelligently identifies and analyzes the metadata in the original data pool in combination with the intermediate data vector, generates an auxiliary processing data set, processes the metadata in the original data pool into a subject domain data table based on the generated auxiliary processing data set, encapsulates the processed subject domain data table into a data service component with a standard API interface, takes each data service component as a reusable data asset, and registers the data service component into a data asset directory for unified management. The application significantly improves the automation and intelligent level of data middle platform construction, effectively guarantees data quality and consistency through metadata intelligent analysis, dynamic modeling and conflict resolution, and solves the problem of data islands.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for assisting in the construction of an omnichannel data platform. Background Technology

[0002] As enterprises deepen their digital transformation, a growing number of business channels (including online stores, offline stores, mobile applications, social media, and third-party platforms) are generating massive amounts of heterogeneous data. This data is typically scattered across different systems, with varying formats and standards, creating severe data silos and making it difficult for enterprises to achieve unified data access, integrated analysis, and value extraction. Therefore, a data platform, as an enterprise-level data sharing and capability reuse platform, is gradually becoming a core infrastructure for enterprises to integrate data resources and support rapid innovation in front-end businesses.

[0003] However, the construction of existing data platforms still faces many challenges. First, traditional data integration methods typically involve customized development for specific channels or systems, lacking unified access standards and specifications, resulting in low access efficiency and poor scalability. Second, in the process of fusion of multi-source heterogeneous data, data cleaning, transformation, and modeling rely heavily on manual experience, which is not only inefficient but also prone to introducing errors, making it difficult to guarantee the consistency and reliability of data quality. Furthermore, existing platform construction methods often lack support for visual management and agile development of data assets, making it difficult for business personnel to quickly understand and utilize the data services within the platform, thus affecting the effectiveness of data-driven business empowerment.

[0004] Therefore, there is an urgent need for a systematic and highly automated auxiliary construction method that can help enterprises efficiently and systematically complete the access, integration and management of omnichannel data, improve the construction efficiency and quality of data middle platform, and ultimately support enterprise data-driven decision-making and business innovation. Summary of the Invention

[0005] This invention provides a method for assisting in the construction of an omnichannel data platform, comprising:

[0006] Step 1: Connect to multiple data sources, convert the collected metadata into intermediate data vectors, and then store them in the original data pool;

[0007] Step 2: Combine intermediate data vectors to intelligently identify and analyze metadata in the original data pool to generate auxiliary processing datasets;

[0008] Step 3: Based on the generated auxiliary processing dataset, process the metadata in the original data pool into subject domain data tables;

[0009] Step 4: Encapsulate the processed subject domain data tables into data service components with standard API interfaces, and register each data service component as a reusable data asset in the data asset catalog for unified management.

[0010] The omnichannel data platform construction method described above, which involves accessing multiple channel data sources and converting the collected metadata into intermediate data vectors before storing them in the original data pool, is specifically divided into the following sub-steps:

[0011] Load the corresponding data connector based on the user-configured data source information;

[0012] The loaded data connector establishes a temporary connection with the target data source and automatically collects its metadata.

[0013] The collected metadata is transformed into intermediate data vectors and persistently stored in the original data pool.

[0014] The omnichannel data platform construction method described above, which combines intermediate data vectors to intelligently identify and analyze metadata in the original data pool, is specifically divided into the following sub-steps:

[0015] Deep features are obtained by fusing intermediate data vectors and knowledge from domain-specific knowledge graphs;

[0016] Clustering of fields in the original data pool based on deep features outputs topic domain clusters;

[0017] Construct a field association subgraph for each subject domain cluster;

[0018] By integrating the output subject domain clusters with the constructed field association graph, a dataset for auxiliary processing is generated.

[0019] The omnichannel data platform construction method described above involves processing the metadata in the original data pool into subject domain data tables based on the generated auxiliary processing dataset. This process is divided into the following sub-steps:

[0020] Use field scoring functions to determine the main fields for each topic domain cluster;

[0021] Use cross-domain field resolution functions to eliminate field redundancy and conflicts between different subject domains;

[0022] The optimal data table structure is dynamically generated for each subject area, converted into a standard data definition language script, and then executed in the target library of the middle platform.

[0023] The present invention also provides an omnichannel data platform auxiliary construction system, including: a data source access module, a metadata analysis module, a subject domain data table generation module, and a data service component encapsulation module;

[0024] The data source access module is used to access data sources from multiple channels, and to convert the collected metadata into intermediate data vectors and store them in the raw data pool.

[0025] The metadata analysis module is used to intelligently identify and analyze the metadata in the original data pool by combining intermediate data vectors, and generate auxiliary processing datasets.

[0026] The subject domain data table generation module is used to process the metadata in the original data pool into subject domain data tables based on the generated auxiliary processing dataset.

[0027] The data service component encapsulation module is used to encapsulate the processed subject domain data tables into data service components with standard API interfaces, and register each data service component as a reusable data asset in the data asset catalog for unified management.

[0028] The beneficial effects achieved by this invention are as follows: It significantly improves the automation and intelligence level of data platform construction, effectively ensures data quality and consistency through intelligent metadata analysis, dynamic modeling and conflict resolution, and solves the data silo problem; at the same time, it automatically encapsulates data tables into standard reusable API services and manages them in a unified manner, realizing agile reuse and efficient governance of data assets, greatly reducing development and maintenance costs, and empowering rapid business innovation. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0030] Figure 1 This is a flowchart of an omnichannel data platform construction method provided in Embodiment 1 of this application. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Example 1

[0033] like Figure 1 As shown, Embodiment 1 of this application provides a method for assisting in the construction of an omnichannel data platform, including:

[0034] Step S110: Connect to multiple data sources, convert the collected metadata into intermediate data vectors, and then store them in the original data pool. This is specifically divided into the following sub-steps:

[0035] Step S111: Load the corresponding data connector based on the user-configured data source information;

[0036] The system receives data source configuration information input by the user through a graphical interface. The configuration information includes data source type, connection address, authentication credentials, and sampling parameters. Based on the data source type, the system automatically matches and loads the corresponding data connector from a pre-set connector library. The data connector supports multiple data source types, including relational databases, non-relational databases, API interfaces, log files, and message queues.

[0037] Step S112: Establish a temporary connection with the target data source using the loaded data connector and automatically collect the metadata therein;

[0038] Using the loaded data connector, a temporary connection is established with the target data source based on the provided connection address and authentication credentials, and a connectivity test is performed. After a successful connection, the available data entities in the data source are automatically detected, and their metadata information is collected. The metadata includes, but is not limited to, data table structure, field names, data types, and constraint information.

[0039] Step S113: Convert the collected metadata into an intermediate data vector and persist it to the original data pool;

[0040] Metadata collected from different data sources is standardized, then uniformly converted into intermediate data vectors and persistently stored in the original data pool. The transformation process from metadata to intermediate data vectors is as follows:

[0041] ① Construct feature groups for each field of the metadata;

[0042] The feature group representation of a single field in metadata is as follows: ,in The word vector for the field name. For the one-hot encoded vector of the data type, A multidimensional binary vector representing the constraints. A statistical vector composed of statistical characteristics of the sample data (proportion of unique values, proportion of missing values, etc.).

[0043] ② Input each of the constructed feature groups into the data transformation formula to obtain the transformed intermediate data vector;

[0044] The data transformation formula is expressed as follows: ,in This is the intermediate data vector after transformation. These are the contribution coefficients of different feature dimensions to the intermediate data vector. These are mapping matrices corresponding to the respective features, obtained through pre-training. The process includes:

[0045] Construct a sample set consisting of the original fields and the standard target fields;

[0046] A dual-tower neural network model was built, using data transformation formulas as field encoders;

[0047] Using contrastive learning as the training paradigm, the optimization objective is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0048] The mapping matrix is ​​iteratively optimized using the backpropagation algorithm until the model converges.

[0049] The mapping matrix obtained from solidified training is used to generate all subsequent intermediate data vectors.

[0050] ③ Associate metadata fields and their transformed intermediate data vectors.

[0051] Step S120: Combine intermediate data vectors to intelligently identify and analyze metadata in the original data pool to generate an auxiliary processing dataset;

[0052] Before performing intelligent identification and analysis on metadata, it is necessary to pre-build a domain-specific knowledge graph based on industry standard data models, the company's existing data dictionary, and business process documents. The construction process is as follows:

[0053] ① Extract entity, relation, and attribute metadata from the collected domain information;

[0054] For structured data, the database metadata (table structure, foreign key constraints, comments) is read directly, and the database tables are automatically mapped to entity types, fields are mapped to attributes, and foreign keys are mapped to relationships. For unstructured or semi-structured data, entity recognition and relationship and attribute extraction are performed based on a pre-trained language model.

[0055] ② After cleaning the extracted three-dimensional data, store it in the native graph database to form a domain knowledge graph;

[0056] Semantic similarity is used to determine whether records from different data sources point to the same entity. If they come from the same entity and have multiple conflicting attribute values, a confidence strategy is formulated based on factors such as the reliability of the data source and timestamps, automatically selecting the most reliable value; other non-conflicting attribute values ​​are merged. After processing, the data is stored in a native graph database to form a domain-specific knowledge graph.

[0057] The metadata in the original data pool is intelligently identified and analyzed by combining intermediate data vectors to generate an auxiliary processing dataset, which is divided into the following sub-steps:

[0058] Step S121: Merge the intermediate data vectors and the knowledge in the domain knowledge graph to obtain deep features;

[0059] First, using field i as the query entity, we retrieve its adjacent subgraph in the domain knowledge graph and generate a semantic embedding vector through a graph attention network; then, we use a lightweight Transformer encoder to embed the intermediate data vector. Contextual semantics are extracted to obtain a context vector; then, multilayer perceptron is used to fuse the context vector and the semantic embedding vector to obtain the deep features of field i. .

[0060] Step S122: Cluster the fields in the original data pool based on deep features to output topic domain clusters;

[0061] For each field i, it is treated as a potential positive sample along with other fields from the same data source, and as a hard negative sample along with fields from different sources but with similar names. A clustering-oriented encoder is trained using contrastive learning loss, which clusters similar fields in the vector space and separates dissimilar fields. The optimal number of clusters k is dynamically determined, and the output topic domain clusters C = {C1, C2, ..., C} are obtained. k}

[0062] Step S123: Construct a field association subgraph for each subject domain cluster;

[0063] Using fields within a single topic domain cluster as nodes, edge weights Through the formula: Calculate and construct a field association subgraph, where i and j represent any field in a single subject domain cluster, and These are the depth features of fields i and j, respectively. This is the cosine similarity calculation function. This is a co-occurrence frequency calculation function used to count the frequency with which fields i and j appear together in actual business data (such as a single response, Jason). This is a naming similarity calculation function used to calculate the string similarity between the names of fields i and j. These are adjustable weighting coefficients.

[0064] To preserve cross-domain relationships between fields, a global field relationship graph needs to be built for all fields, constructed in the same way as the subgraph.

[0065] Step S124: Integrate the output topic domain clusters with the constructed field association graph to obtain the generated auxiliary processing dataset;

[0066] The generated auxiliary processing dataset includes a list of subject domains, a list of fields under each domain, a global field relationship graph, and field relationship subgraphs under each domain.

[0067] Step S130: Based on the generated auxiliary processing dataset, process the metadata in the original data pool into a subject domain data table;

[0068] Based on the generated auxiliary processing dataset, the metadata in the original data pool is automatically and intelligently constructed into subject domain data tables, which is divided into the following sub-steps:

[0069] Step S131: Use the field scoring function to determine the main field of each subject domain cluster;

[0070] The field scoring function is used to calculate the score of each field within a subject domain cluster, and the top 20% of fields with the highest scores are selected as the main field set for that subject domain, used to construct the core structure of the data table; the field scoring function... Represented as: Where i represents any field within the current subject domain cluster, This indicates that field i is in the global field association graph. The degree in the middle, This represents the edge weights of fields i and j within the subgraph associated with the fields of the current topic domain. , Let j be the set of adjacent fields within the field association subgraph. This indicates the total number of fields in the adjacent field set. Represents the deep feature vector of field i The cosine similarity between the semantic vector of the entity E corresponding to the current topic domain in the domain knowledge graph and the semantic vector of the entity E. This is an adjustable parameter.

[0071] Step S132: Use cross-domain field resolution functions to eliminate field redundancy and conflicts between different subject domains;

[0072] To avoid field redundancy and conflicts between data tables of different subject domains, this application constructs a cross-domain field parsing function to determine the subject domain to which a field existing in multiple subject domain clusters ultimately belongs. Represented as: ,in For subject area indexing, It is the collection of all subject domain clusters containing field i. For field i in the subject domain The rating in the middle, For field i to the subject domain The specificity of field i, i.e., the relationship between field i and the subject domain. The average association strength of the main field set, This is a weighting factor.

[0073] Field i was ultimately divided into Within a subject domain, only one soft link (foreign key association) is retained in other subject domains, thereby achieving both the uniqueness of physical storage and the completeness of logical association.

[0074] Step S133: Dynamically generate the optimal data table structure for each subject area, convert it into a standard data definition language script, and then execute it in the target library of the middle platform;

[0075] Using the selected set of primary fields as the seed table structure, calculate the target value of each candidate field (other fields not included in the seed table structure) in the current subject domain cluster after adding it to the seed table structure. Select the candidate field with the smallest target value and add it to the seed table structure. Then, calculate the target value of each candidate field after adding it to the current seed table structure again, and select the candidate field with the smallest target value again. Repeat this process until the preset number of fields is reached, and output the current seed table structure as the final data table structure of the current subject domain.

[0076] Function for calculating the target value Represented as: ,in For the current seed table structure, Returns the average association strength between field i and all fields in the main field set P. Assign the edge weights between fields i and j in the field association subgraph of the current topic domain. is the regularization coefficient.

[0077] Using custom rules or templates, the output topic domain data table structure is converted into a DDL script for the middleware target library. After execution, the corresponding physical table will be automatically created in the library.

[0078] Step S140: Encapsulate the processed subject domain data table into a data service component with a standard API interface, and register each data service component as a reusable data asset in the data asset catalog for unified management;

[0079] Define a standard RESTful API interface for each generated subject domain data table, including but not limited to common data operation interfaces such as single query, batch query, pagination query, and conditional filtering; the input parameters and output format (JSON / XML) of the API follow the predefined specifications; then, encapsulate the subject domain data table and its corresponding API interface into an independent data service component, publish it to the data platform's runtime environment, and provide access addresses to the outside world; the published data service component is registered in the data asset catalog as a reusable data asset to achieve unified monitoring, version management, and access control, ensuring the controllability and security of data assets.

[0080] Example 2

[0081] This application provides a second embodiment of a multi-channel data middle platform auxiliary construction system, including: a data source access module, a metadata analysis module, a subject domain data table generation module, and a data service component encapsulation module;

[0082] (1) Data source access module, used to access multiple data sources, convert the collected metadata into intermediate data vectors and store them in the original data pool; specifically including: data connector loading submodule, metadata collection submodule, and metadata conversion submodule;

[0083] 1. The data connector loading submodule is used to load the corresponding data connector based on the data source information configured by the user.

[0084] 2. Metadata collection submodule, used to establish a temporary connection with the target data source using the loaded data connector, and automatically collect the metadata therein;

[0085] 3. Metadata Transformation Submodule: This module is used to transform the collected metadata into intermediate data vectors and persistently store them in the original data pool.

[0086] (2) Metadata analysis module, which is used to intelligently identify and analyze the metadata in the original data pool by combining intermediate data vectors to generate auxiliary processing datasets; specifically including: domain knowledge graph construction submodule, deep feature generation submodule, topic domain cluster output submodule, field association graph construction submodule;

[0087] 1. Domain-specific knowledge graph construction submodule, used to construct a domain-specific knowledge graph based on data information within the business domain;

[0088] 2. Deep Feature Generation Submodule, used to fuse intermediate data vectors and knowledge from the domain knowledge graph to obtain deep features;

[0089] 3. The topic domain cluster output submodule is used to cluster fields in the original data pool based on deep features and output topic domain clusters;

[0090] 4. Field Association Graph Construction Submodule: This module is used to construct a global field association graph and to build a field association subgraph for each subject domain cluster.

[0091] (3) Subject domain data table generation module, which is used to process the metadata in the original data pool into subject domain data tables based on the generated auxiliary processing dataset; specifically including: main field determination submodule, redundancy elimination submodule, physical table creation submodule;

[0092] 1. The main field determination submodule is used to determine the main field of each subject domain cluster using field scoring functions;

[0093] 2. Redundancy elimination submodule, used to eliminate field redundancy and conflicts between different subject domains using cross-domain field resolution functions;

[0094] 3. The physical table creation submodule is used to dynamically generate the optimal data table structure for each subject domain, convert it into a standard data definition language script, and then execute it in the target library of the middle platform.

[0095] (4) Data service component encapsulation module, which is used to encapsulate the processed subject domain data table into a data service component with a standard API interface, and register each data service component as a reusable data asset in the data asset catalog for unified management.

[0096] Corresponding to the above embodiments, the present invention provides a computer storage medium, including: at least one memory and at least one processor;

[0097] The memory is used to store one or more program instructions;

[0098] A processor is used to run one or more program instructions to execute a method for building an omnichannel data platform.

[0099] Corresponding to the above embodiments, this embodiment of the invention provides a computer-readable storage medium containing one or more program instructions, which are executed by a processor to provide a method for assisting in the construction of an omnichannel data platform.

[0100] The embodiments disclosed in this invention provide a computer-readable storage medium storing computer program instructions, which, when executed on a computer, cause the computer to perform the aforementioned method for assisting in the construction of an omnichannel data platform.

[0101] In this embodiment of the invention, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0102] The various methods, steps, and logic diagrams disclosed in the embodiments of this invention can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor reads information from the storage medium and, in conjunction with its hardware, completes the steps of the above methods.

[0103] The storage medium can be memory, such as volatile memory or non-volatile memory, or may include both volatile and non-volatile memory.

[0104] Among them, non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory.

[0105] Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0106] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0107] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using a combination of hardware and software. When applied as software, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0108] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for assisting in building a full-channel data hub, characterized in that, The method comprises the following steps: Step 1, access multi-channel data sources, and store the collected metadata into an original data pool after being converted into intermediate data vectors; The conversion process of the metadata into the intermediate data vectors comprises the following steps: ① constructing feature groups for each field of the metadata; The feature set of the metadata single field is represented as: wherein is a word vector of field name, is a one-hot encoding vector of data type, is a multi-dimensional binary vector of constraint conditions, is a statistical vector composed of statistical features of sample data; ② inputting each constructed feature group into a data conversion formula to obtain the converted intermediate data vector; The data conversion formula is expressed as: wherein is the converted intermediate data vector, is the contribution coefficient of different feature dimensions to the intermediate data vector, is the mapping matrix corresponding to the feature, which is obtained by pre-training, and the process includes: constructing a sample set composed of original fields and standard target fields; building a double-tower neural network model, and using the data conversion formula as a field encoder; taking contrastive learning as a training paradigm, and taking maximizing the similarity of positive samples and minimizing the similarity of negative samples as optimization objectives; iteratively optimizing the mapping matrix through an error back propagation algorithm until the model converges; solidifying the mapping matrix obtained through the training for generating all subsequent intermediate data vectors; ③ associating the metadata fields and the converted intermediate data vectors; Step 2, intelligently identifying and analyzing the metadata in the original data pool in combination with the intermediate data vectors to generate an auxiliary processing data set, which comprises the following sub-steps: fusing the intermediate data vectors and the knowledge in the field knowledge graph to obtain deep features; clustering the fields in the original data pool based on the deep features to output theme domain clusters; constructing a field association subgraph for each theme domain cluster; integrating the output theme domain clusters and the constructed field association subgraphs to generate the auxiliary processing data set; Step 3, processing the metadata in the original data pool into theme domain data tables based on the generated auxiliary processing data set; The field scoring function is used to calculate the score of each field within a subject domain cluster, and the top 20% of fields with the highest scores are selected as the main field set for that subject domain, which is used to construct the core structure of the data table; the field scoring function... Represented as: Where i represents any field within the current subject domain cluster, This indicates that field i is in the global field association graph. The degree in the middle, This represents the edge weights of fields i and j within the subgraph associated with the fields of the current topic domain. , Let j be the set of adjacent fields within the field association subgraph. This indicates the total number of fields in the adjacent field set. Represents the deep feature vector of field i The cosine similarity between the semantic vector of the entity E corresponding to the current topic domain in the domain knowledge graph and the semantic vector of the entity E. These are adjustable parameters; Step 4, encapsulating the processed theme domain data tables into data service components with standard API interfaces, and registering each data service component as a reusable data asset in a data asset directory for unified management.

2. The full-channel data hub assisted construction method according to claim 1, characterized in that, Accessing multi-channel data sources, and storing the collected metadata into an original data pool after being converted into intermediate data vectors, which comprises the following sub-steps: loading a corresponding data connector according to the data source information configured by a user; establishing a temporary connection with a target data source by using the loaded data connector, and automatically collecting metadata therein; converting the collected metadata into intermediate data vectors and persistently storing them into the original data pool.

3. The method of claim 2, wherein the full channel data hub assisted construction is characterized by, The construction of the field knowledge graph comprises the following sub-steps: extracting entity, relation and attribute triadic data from collected field information; cleaning the extracted triadic data and storing them into a native graph database to form the field knowledge graph.

4. The full-channel data hub assisted construction method according to claim 3, characterized in that, Processing the metadata in the original data pool into theme domain data tables based on the generated auxiliary processing data set, which comprises the following sub-steps: determining the main fields of each theme domain cluster by using a field scoring function; eliminating the field redundancy and conflicts between different theme domains by using a cross-domain field parsing function; dynamically generating the optimal data table structure for each theme domain, and executing the converted standard data definition language script in a target library of a middle platform.

5. A full-channel data middle platform assisted construction system, characterized in that, The method is used for assisting in constructing a full-channel data middle platform, and comprises a data source access module, a metadata analysis module, a theme domain data table generation module and a data service component encapsulation module. The data source access module is configured to access multi-channel data sources, and store the collected metadata into an original data pool after converting the metadata into intermediate data vectors. The metadata analysis module is configured to intelligently identify and analyze the metadata in the original data pool in combination with the intermediate data vectors, and generate an auxiliary processing data set. The theme domain data table generation module is configured to process the metadata in the original data pool into theme domain data tables based on the generated auxiliary processing data set. The data service component packaging module is configured to package the processed theme domain data tables into data service components with standard API interfaces, and register each data service component as a reusable data asset into a data asset directory for unified management.

6. The full channel data hub assisted build system according to claim 5, wherein, The data source access module specifically includes a data connector loading submodule, a metadata collection submodule, and a metadata conversion submodule. The data connector loading submodule is configured to load corresponding data connectors according to user-configured data source information. The metadata collection submodule is configured to establish a temporary connection with a target data source by using the loaded data connectors, and automatically collect metadata therein. The metadata conversion submodule is configured to convert the collected metadata into intermediate data vectors and persistently store the intermediate data vectors into the original data pool.

7. The full channel data hub assisted build system according to claim 5, wherein, The metadata analysis module specifically includes an intra-domain knowledge graph construction submodule, a deep feature generation submodule, a theme domain cluster output submodule, and a field association graph construction submodule. The intra-domain knowledge graph construction submodule is configured to construct an intra-domain knowledge graph based on data information in a business domain. The deep feature generation submodule is configured to fuse the intermediate data vectors and knowledge in the intra-domain knowledge graph to obtain deep features. The theme domain cluster output submodule is configured to cluster fields in the original data pool based on the deep features, and output theme domain clusters. The field association graph construction submodule is configured to construct a global field association graph, and construct a field association subgraph for each theme domain cluster.

8. A computer storage medium, characterized in that The system includes at least one memory and at least one processor. The memory is configured to store one or more program instructions. The processor is configured to run the one or more program instructions to execute the full-channel data platform auxiliary construction method according to any one of claims 1-4. ​

Images