Machine learning-based heterogeneous data processing methods, electronic devices, and storage media

Abstract

This application relates to the field of data artificial intelligence processing technology, specifically to a heterogeneous data processing method, electronic device, and storage medium based on machine learning. This application generates a unified dense feature vector representation structure based on a machine learning mechanism of self-supervised representation learning. It constructs a multi-source heterogeneous data query system driven by a dynamic knowledge graph, executed by proactive data weaving, and outputting a unified representation of heterogeneous data. This system enables adaptive business connections between heterogeneous data and efficiently meets users' explicit or potential query needs. This application can overcome the limitations of traditional methods that rely on manually formulated hard matching rules, improve the automated management capabilities of ultra-large-scale asset data, and enhance the performance of data interoperability and seamless integration.

Description

Technical Field

[0001] This invention relates to the field of data artificial intelligence processing technology, specifically to a heterogeneous data processing method, electronic device, and storage medium based on machine learning. Background Technology

[0002] For large-scale asset management, the numerous business departments involved and the complexity of management make it difficult to standardize asset classification criteria, coding rules, and field names. This results in poor data integration and compatibility, with asset data from different business departments typically scattered across their respective systems, creating data silos. Currently, achieving comprehensive integrated management relies heavily on manual data entry and verification, which is labor-intensive, prone to errors, and difficult to control in terms of cost. Summary of the Invention

[0003] The purpose of this application is to provide a heterogeneous data processing method, electronic device and storage medium based on machine learning, which solves the technical problems of difficulty in interoperability and mutual recognition of asset data management in the prior art, and the need for a lot of time to manually integrate and verify data.

[0004] To solve the above-mentioned technical problems, this application adopts the following technical solution.

[0005] Firstly, this application provides a heterogeneous data processing method based on machine learning, including:

[0006] A dynamic knowledge graph for query reference is generated, which generates a graph structure based at least on first asset data from a first data source and second asset data from a second data source. The graph structure includes at least a first metadata vector representation in the first asset data, a second metadata vector representation in the second asset data, and an association index. The first metadata includes at least a first asset ontology and a first business tag, and the second metadata includes at least a second asset ontology and a second business tag. The association index establishes a connection based on the vector similarity between the first metadata and the second metadata.

[0007] In response to the user's query request, the first metadata of the first data source and the second metadata of the second data source are queried according to the associated index, and the first metadata and the second metadata are merged and processed according to the query request to meet the corresponding output format.

[0008] In one alternative implementation of the first aspect, the first asset data and the second asset data are in a converted standard intermediate format.

[0009] In one alternative implementation of the first aspect, the standard intermediate format is JSON or XML.

[0010] In one alternative embodiment of the first aspect, the method includes:

[0011] Configure a first execution program for generating the dynamic knowledge graph and a second execution program for data querying;

[0012] The runtime resources of the first and second executable programs are dynamically allocated based on the working status of at least one of the real-time CPU, memory, network bandwidth, and disk I / O load on which the first and second executable programs depend.

[0013] In one alternative embodiment of the first aspect, the priority of the second executor is higher than the priority of the first executor.

[0014] In one alternative embodiment of the first aspect, the method includes:

[0015] Periodically scan at least one of the following: table structure, column definitions, data types, index information, statistical information, and data distribution of the first data source and the second data source;

[0016] When the data difference exceeds a first threshold, an update to the dynamic knowledge graph is triggered.

[0017] In one alternative embodiment of the first aspect, the method includes:

[0018] In response to an update to the first asset data, the first asset data is made to include the first asset data at a first time point and the first asset data at a second time point to construct context information.

[0019] In one alternative embodiment of the first aspect, the step of making the first asset data include first asset data at a first time point and first asset data at a second time point includes:

[0020] The corresponding equipment is in its first maintenance state at the first time point and in its second maintenance state at the second time point;

[0021] The health status of the corresponding device is determined based on the contextual timing information formed by the first maintenance status and the second maintenance status.

[0022] In one alternative embodiment of the first aspect, the method includes:

[0023] Obtain an asset procurement list for comparison with the asset ontology. The first asset ontology and the second asset ontology are extracted based on the asset procurement list using a pre-trained model.

[0024] In one alternative implementation of the first aspect, the first asset ontology and the second asset ontology are extracted based on a pre-trained model in conjunction with the asset procurement list, including:

[0025] If the names of the first asset entity and the second asset entity are inconsistent, the corresponding asset entity name in the asset procurement list shall prevail.

[0026] In one alternative implementation of the first aspect, the user's query request is natural language input, and the user's query intent is determined based on the natural language input to generate an execution strategy that performs the query at least between the first data source and the second data source.

[0027] In an alternative implementation of the first aspect, determining the user's query intent based on the natural language input includes:

[0028] Semantic parsing is performed using natural language processing to generate a demand vector corresponding to the query requirements;

[0029] Calculate the semantic similarity between the demand vector and each node of the dynamic knowledge graph to determine the matching asset ontology.

[0030] In one optional implementation of the first aspect, querying the first metadata of the first data source and the second metadata of the second data source according to the association index includes:

[0031] Obtain the operation type, operation function, and / or indexing capability of the first data source and the second data source;

[0032] The query requirements are decomposed into the first data source and the second data source according to the query requirements, so that the first data source and the second data source return only the matching data.

[0033] In one optional implementation of the first aspect, querying the first metadata of the first data source and the second metadata of the second data source according to the association index includes:

[0034] In response to the existence of duplicate data segments between the first asset data and the second asset data, the data source for returning the duplicate data segments is selected based on the communication service quality of the first data source and the second data source.

[0035] In one alternative embodiment of the first aspect, the method includes:

[0036] Analyze the user's historical query needs to predict the user's query preferences;

[0037] Based on the query preferences, the returned data from the first data source and the second data source are obtained in advance and then merged and processed.

[0038] In one alternative implementation of the first aspect, analyzing the user's historical query needs to predict the user's query preferences includes:

[0039] Based on the user's query time distribution, predict the user's future query time;

[0040] Based on the distribution of the user's query content, predict the user's future query content;

[0041] The step of pre-obtaining and merging the returned data from the first data source and the second data source according to the query preference includes:

[0042] The future query content is generated and automatically pushed to the user at the future query time.

[0043] In a second aspect, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the heterogeneous data processing method based on machine learning described in the first aspect.

[0044] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the heterogeneous data processing method based on machine learning described in the first aspect.

[0045] Compared with existing technologies, this application generates a unified dense feature vector representation structure based on a machine learning mechanism of self-supervised representation learning. It constructs a multi-source heterogeneous data query system driven by a dynamic knowledge graph, executed by proactive data weaving, and ultimately outputting a unified representation of heterogeneous data. This system enables adaptive business connections between heterogeneous data and efficiently meets users' explicit or potential query needs. This application overcomes the limitations of traditional methods that rely on manually defined hard matching rules, improves the automated management capabilities of ultra-large-scale asset data, and enhances the performance of data interoperability and seamless integration. Attached Figure Description

[0046] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the technical solution will be briefly introduced below. Obviously, the drawings described below are merely some examples recorded in this application, and those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0047] Figure 1 The following are schematic diagrams of heterogeneous data processing architectures based on machine learning, as shown in some examples of this application.

[0048] Figure 2 Here are flowcharts of machine learning-based heterogeneous data processing methods in some examples of this application. Detailed Implementation

[0049] The present application will be described in detail below with reference to the accompanying drawings. However, the description is only a few examples recorded in the present application and does not limit the present application. Any changes in structure, method or function made by those skilled in the art based on these examples are included within the protection scope of the present application.

[0050] It should be noted that while the same labels or markers may be used in different examples, these do not represent an absolute structural or functional relationship. Furthermore, the use of terms such as "first," "second," etc., in the examples is merely for descriptive convenience and does not represent an absolute structural or functional distinction, nor should it be interpreted as indicating or implying relative importance or the number of corresponding objects. Unless otherwise specified, "at least one" in the description refers to one or more, and "more than one" refers to two or more.

[0051] Furthermore, when representing features, the character " / " can indicate an OR relationship between the preceding and following objects; for example, A / B can be represented as A or B. When representing operations, the character " / " can indicate a division relationship between the preceding and following objects; for example, C=A / B can be represented as C equal to A divided by B. Moreover, the use of "and / or" in different examples is merely to describe the relationship between the preceding and following objects, which can include three cases: for example, A and / or B can be represented as A existing alone, B existing alone, or both A and B existing simultaneously.

[0052] Multi-source heterogeneous data refers to the situation in large-scale asset data management where multiple data sources maintain asset data with different dimensions, and the data structures of different data sources are usually inconsistent. Therefore, multi-source heterogeneous data integration solutions are generally rigid and difficult to adapt to dynamic changes. The traditional "human-finds-data" model is inefficient and slow to respond. Fragmented data is difficult to understand and represent effectively. Existing data weaving technologies suffer from insufficient intelligence in data source processing, and the lack of performance awareness and self-optimization feedback mechanisms during the data weaving process prevents the formation of a closed-loop iterative system of "collection-weaving-service-optimization".

[0053] In some examples, such as Figure 1As shown, asset data may exist from multiple data sources due to the management activities of different departments. For example, the finance department might have one table for inventorying asset value, the maintenance department for managing daily inspections, and the user department for recording routine logs. Consequently, there could be a first data source, a second data source, a third data source, or even more. However, in this example, the data sources are not simply managed independently by each management department. This would inevitably increase communication and management costs due to inconsistent standards, and queries would not cover the entire system. Nor is it simply a matter of forcibly integrating data from different data sources. This would inevitably involve integration efficiency issues, and large-scale data migration and redundant storage would incur additional costs and would not be able to adapt to the dynamic updates of different data sources. Therefore, this example sets up a query virtualization layer between the query user and multiple data sources. This query virtualization layer acts as an intermediary layer, serving as an interface for receiving user queries. It not only responds to user query requests but can also predict potential data needs based on query records. Furthermore, it configures dynamic adapters to connect to various types of data sources, enabling not only queries to data sources that meet the requirements but also the construction and updating of a dynamic knowledge graph on top of the source data, as detailed below. The query virtualization layer can be configured in a central server, which can connect to servers maintained by multiple data sources. Optionally, data from multiple data sources can also be directly stored in this central server, meaning a system has only one or a cluster of central servers, with access control implemented according to management departments. Alternatively, the query virtualization layer can also be configured sharedly on a server maintained by a specific data source, executing specific programs within a specific permission domain.

[0054] In some examples, such as Figure 2 As shown, the heterogeneous data processing method based on machine learning specifically includes the following steps:

[0055] Step S1: Dynamic Data Access and Preprocessing. To meet the data management and query needs of various departments, an asset data system can be built for all departments. As mentioned above, the asset data system can establish a virtual query structure between users and specific data sources through a query virtualization layer, connect databases from multiple data sources, configure dynamic adapters to connect various types of data sources, and process the raw data to convert it into a unified standard intermediate format so that it can be easily recognized and processed by the program. For example, JSON (JavaScript Object Notation) or XML (Extensible Markup Language) formats can be used to extract key fields from structured tables or unstructured text and encapsulate them one by one using the above formats, so that the asset data has a clear asset ontology and business tag data, which is easy to identify and easy to statistically analyze. Furthermore, the system can periodically scan the table structure, column definitions, data types, index information, statistical information, and data distribution metadata of each data source, such as a daily or hourly cycle. This allows for timely location of data changes in specific data sources, and can automatically trigger relevant updates in the query virtualization layer after detecting metadata changes. This ensures that the query of the query virtualization layer and the management of the underlying data source are not disconnected.

[0056] Step S2: Dynamic Knowledge Graph Construction and Semantic Injection. Based on the data access preprocessing, the asset data system can utilize the first execution program to construct a dynamic knowledge graph to form a structured knowledge network. This achieves a fundamental transformation of multi-source heterogeneous data from "fragmented storage" to "knowledge-based organization." Essentially, it does not simply copy the data from the data source to the query virtualization layer, but rather establishes an overview of all data sources in the query virtualization layer to guide the query process and quickly extract data from the corresponding data sources. Correspondingly, information extraction techniques based on pre-trained language models (such as BERT or GPT models) are used to automatically extract core entities, entity attributes, and semantic relationships between entities from the standardized dataset, thereby determining the asset ontology and business tags mentioned in the example above. For example, if the first data source contains first asset data and the second data source contains second asset data, and since the first and second asset data are in a standardized intermediate format after conversion, a language model can be used to identify the metadata within them. This includes first metadata such as the first asset ontology and first business tag, and second metadata such as the second asset ontology and second business tag. In a specific example, each asset ontology can be extracted to match more business tags. Taking the asset ontology of central air conditioning as an example, it includes its associated machine room, equipment status, asset value, and commissioning time. Furthermore, based on data processing, using the extracted entities as nodes and entity semantic relationships as edges, an initial knowledge graph is constructed using a graph database, and graph embedding representation learning technology is introduced to generate low-dimensional vector representations of entity nodes. The purpose of knowledge graph generation is to obtain the metadata vector representations of all asset ontologs, thereby enabling fuzzy matching of data through vector computation. For example, the same asset ontology from different data sources can be associated through vector similarity (cosine similarity algorithm, etc.), without requiring complete integration and storage of the source data. Optionally, asset ontology extraction can also be compared with the asset procurement list, as the asset procurement list is the source of asset ontology management for all departments. By injecting rules, a higher-level semantic concept system can be added to the knowledge graph, generating semantic web ontology data. In a specific example, cross-data source entity parsing and semantic disambiguation can be performed based on the asset procurement list. For instance, if the first asset ontology from the first data source and the second asset ontology from the second data source have different names, but vector similarity determines that the first and second asset ontology represent the same asset, then the names in the asset procurement list can be used to replace the original names of the first and second asset ontology.

[0057] As described above, when cross-data sources include different dimensions of the same asset ontology, at least the first metadata vector representation of the first asset data can be extracted from the first asset data of the first data source, and at least the second metadata vector representation of the second asset data can be extracted from the second asset data of the second data source. Furthermore, the association index between the first and second metadata vector representations can be determined based on the vector similarity between them, thereby establishing a semantic association mapping network between ontology data and the instance-layer structured knowledge graph. This achieves unified integration of multi-source data at the semantic level, meaning that association indexes can be used to locate data of the same asset ontology across multiple data sources, enabling the targeting of extracted data when responding to query requests. In practical applications, multiple sets of association indexes can be obtained through vector computation to correspond to multiple actual asset ontologs. The query process will be detailed below.

[0058] Since queries implemented at the query virtualization layer are not directly initiated to various data sources, but are based on a pre-maintained dynamic knowledge graph, to ensure consistency between the dynamic knowledge graph and data from multiple data sources, when the asset data system detects changes in the underlying data sources or detects new data entering the database, it automatically triggers an incremental update process for the knowledge graph, updating the corresponding entities, relationships, and attributes to maintain consistency between the low-dimensional vector space and the semantic space. Specifically, a first threshold can be set; when the data difference exceeds this threshold—for example, if a new asset ontology is added to an individual data source—an update to the dynamic knowledge graph needs to be triggered to cover the new asset ontology. This is achieved through a first execution program that updates the graph for the changed data to maintain consistency with the underlying data sources. Preferably, in response to data changes, the data before and after the change is also labeled. Taking the first asset data as an example, the first asset data could include the first asset data at a first time point and the first asset data at a second time point. By introducing a time window mechanism, contextual labels are added to knowledge fragments along the time dimension, constructing a dynamically evolving knowledge graph containing temporal information. Specifically, the maintenance status of air conditioning equipment may change at different times due to maintenance, use, and other reasons, resulting in data changes. In this example, the maintenance status at different time points will be recorded according to time-series characteristics. This allows us to determine the health status of the air conditioning equipment based on time-series analysis, and the maintenance status can be incorporated into a knowledge graph for subsequent querying.

[0059] Preferably, the system can also integrate a feedback collection module in conjunction with the multimodal service interface to proactively collect user satisfaction evaluations of the pushed data, as well as data patterns where users actively request data through the query interface without actively pushing the data. This feedback data is incorporated into the training process of the knowledge graph construction, serving as a supervisory signal for user preference learning. Furthermore, the asset data system can periodically trigger incremental updates to the knowledge graph based on new user feedback and user query behavior data. Specifically, it utilizes distribution offset detection methods such as KL divergence to monitor the degree of change in the knowledge graph structure in real time. When the degree of change exceeds a preset threshold, it automatically initiates incremental reconstruction of the knowledge graph and retraining of the embedded representations, ensuring the accuracy and timeliness of knowledge-driven processes.

[0060] Step S3: Proactive Data Weaving Supported by Dynamic Knowledge Graphs. The focus of data weaving is not simply aggregating data from different data sources and then executing queries, but rather responding to query requests by selectively extracting data and then merging only the matching data. Correspondingly, the second execution program in the asset data system senses and understands the user's data needs in real time, proactively discovering and combining data units with relationships from various sources and formats of heterogeneous data. In a specific example, accepting user query requests can support querying multiple terminals connected to the virtualization layer, such as computers in various departments, thus receiving specific query requests from multiple channels. Query requests can be structured, for example, by limiting the scope of the query request through form completion. Optionally, the input can also be unstructured, such as natural language queries. Users can directly express text input like "I need all the data for a specific device." Natural language processing (NLP) techniques are used to semantically analyze the user's query, extracting key entities, concepts, conditions, and intents to generate a user demand vector. The semantic similarity between the demand vector and each node in the dynamic knowledge graph is calculated. Specifically, entity linking technology is used to link keywords in the query demand to corresponding entity nodes in the knowledge graph, identifying the set of entities and relationships related to the demand's semantics. This determines the matching asset ontology and, based on the associated indexes of the asset ontology, the data source to initiate the query. The execution strategy for initiating the query data source includes not only the specific query object but also the query method and query load management.

[0061] Accordingly, after understanding the query requirements, the query virtualization layer does not directly execute the query through the second execution program. Instead, it establishes connections with the corresponding data sources based on the target data source list and the association paths and conditions between them. Taking a specific first and second data source as an example, it obtains the operation types, operation functions, and / or indexing capabilities of the first and second data sources. It then decomposes the query requirements into sub-requirements that each data source needs to execute. By configuring the aforementioned operation types and functions, these sub-requirements are issued. The specific query process is then performed directly by the corresponding data source according to its own capabilities, thus returning only the matching data, filtering out mismatched data, reducing ineffective large-scale data migration, and only pulling the minimum necessary dataset to the virtualization layer for processing. After receiving the returned data, the second execution program only combines it according to the query requirements. For example, if different fields have sorting requirements, especially if the data for different fields comes from different data sources, the second execution program needs to complete the final merging operation to meet the user's desired output format. Furthermore, the query process will be optimized based on the capabilities of different data sources. For example, if it is determined that data needs to be retrieved from both the first and second data sources, but different departments have different maintenance habits, there may be duplicate data segments maintained by multiple departments. To meet the query requirements, the second execution program only needs to obtain one copy to complete the merging operation. In this case, the data source with better communication service quality can be selected to return duplicate data segments based on the communication service quality of the first and second data sources. The data source with relatively poor communication service quality will minimize data transmission, transmitting only the necessary data, thereby achieving the optimal transmission strategy and ensuring that the second execution program can obtain the required data as quickly as possible. The communication service quality of the data sources can be determined through periodic scanning and analysis. Correspondingly, performance monitoring points can be deployed at key processing nodes throughout the system to collect key performance indicators such as data scanning latency, knowledge graph construction time, query optimization time, data retrieval latency, and fusion processing throughput in real time. Among these, the priority of the corresponding data sources can be ranked according to the relevant indicators of different data sources.

[0062] In some examples, in addition to responding to users' actual query needs, the asset data system automatically identifies high-frequency queries and hot data partitions based on historical query patterns and data access frequency. It intelligently recommends and creates partial query views to accelerate query response, such as pre-defined programs that directly retrieve data from specific data sources, or database indexes based on specific data content from specific data sources. Furthermore, it can analyze users' historical query needs to determine their common query content and habits, thereby predicting user query preferences. Query preferences can include predicting future query times based on the distribution of user query times and predicting future query content based on the distribution of user query content. This allows for the pre-acquisition and merging of data based on these query preferences, presenting the merged data immediately at the predicted future query time, improving data query speed and user experience. For example, consider the equipment routine report. Based on user query preferences, it's determined that the query is performed every Tuesday. The equipment routine report includes data from both the maintenance and user departments. Therefore, by predicting and pre-acquiring data from multiple departments before Tuesday, the generated equipment routine report can be pre-stored. When the predicted Tuesday arrives or the user's query request is issued, the pre-stored equipment routine report is directly invoked.

[0063] As described above, the first executable program used for generating the dynamic knowledge graph and the second executable program used for data querying can be processed in parallel or executed in a time-sharing manner according to the allocation of runtime resources. In some examples, the second executable program has a higher priority than the first executable program, that is, it prioritizes meeting the user's data query needs, and the first executable program can silently update the corresponding dynamic knowledge graph during periods without queries. Specifically, by obtaining the real-time hardware status of the servers where the first and second executable programs reside, the runtime resources of the first and second executable programs can be dynamically allocated based on at least one of the real-time working states of CPU, memory, network bandwidth, and disk I / O load, ensuring that the running efficiency of the first and second executable programs meets the actual system requirements.

[0064] Step S4: Data Fusion and Unified Representation. After completing the proactive data weaving, the asset data system, in addition to merging data, intelligently fuses the acquired multi-source heterogeneous data to form a unified representation and output format for direct consumption by upper-layer applications, satisfying the final output form. In a specific example, the system receives the woven multi-source heterogeneous data and uses a graph embedding method based on the semantic attributes of knowledge graph nodes to map entities and attributes from different data sources to a unified semantic space, performing entity alignment and attribute fusion operations. For the fusion of real-time streaming data and static data, the system uses windowed streaming processing technology to dynamically associate streaming events with entities in the static knowledge graph, achieving real-time fusion of mixed modal data. Furthermore, the asset data system converts the fused data into various standard output formats, including structured table views, knowledge graph views, and semantic natural language views.

[0065] Among them, the structured table view is to integrate multi-source heterogeneous data and organize it into a standard row list table format with unified fields, unified standards, and neat alignment, as shown in the table below.

[0066] Air-cooled central air conditioning Computer Room 1 normal 2024-03

[0067] A knowledge graph view uses entities as nodes and business relationships as edges to display the inherent relationships of data in a graphical network. For example, it can represent the relationships between assets, data centers, departments, equipment, and maintenance events. Correspondingly, it could be a node like: Central Air Conditioning → Deployed in → Data Center No. 1; Attributes like: Status Normal / 2024 Production. It can be applied to semantic retrieval and association exploration in other systems.

[0068] Semantic Natural Language View is a natural language description data generated based on knowledge work orders. It automatically generates a fluent business natural language description from the integrated structured asset data. It is not a table or a graph, but a sentence or a paragraph of text. For example, a wind-cooled central air conditioner was put into operation in Data Center No. 1 in March 2024. The equipment is currently operating normally and belongs to the category of data center power control. This output format is easier for users to understand and matches the user's natural language input query.

[0069] In some examples, the asset data system includes application-oriented multimodal service interfaces that provide a unified external data service interface. This interface delivers uniformly represented data to upper-layer application systems in a flexible manner, such as supporting data access interfaces that support multiple query methods including standard SQL, natural language, and RESTful APIs. It receives data requests from users or applications, shielding them from the heterogeneity of the underlying data sources. Based on historical query patterns, user behavior analysis, and user preference modeling provided by the knowledge graph, the system proactively pushes asset data relevant to users' current business scenarios. The asset data system also provides a graphical data exploration interface that displays a panoramic view of the knowledge graph, supporting interactive operations such as node expansion, relationship exploration, and semantic search.

[0070] Based on the above examples, the technical solutions involved in this application can be directly embodied in hardware, software modules executed by a control unit, or a combination of both, i.e., one or more steps and / or combinations of one or more steps. These can correspond to various software modules in a computer program flow, or to various hardware modules, such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or any suitable combination thereof. For ease of description, the above description divides the functions into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware components.

[0071] Through the above description of examples, those skilled in the art can clearly understand that this application can be implemented using software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution involved in this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This software is executed by a microcontroller unit and, depending on the required configuration, can include one or more microcontroller units of any type, including but not limited to microcontroller units, microcontrollers, DSPs (Digital Signal Processors), or any combination thereof. The software is stored in memory, such as volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory, flash memory), or any combination thereof.

[0072] In summary, this application generates a unified, dense feature vector representation structure based on a self-supervised representation learning machine learning mechanism. It constructs a multi-source heterogeneous data query system driven by a dynamic knowledge graph, executed through proactive data weaving, and ultimately outputting a unified representation of heterogeneous data. This system enables adaptive business connections between heterogeneous data and efficiently meets users' explicit or potential query needs. This application overcomes the limitations of traditional methods that rely on manually defined hard matching rules, improves the automated management capabilities of ultra-large-scale asset data, and enhances the performance of data interoperability and seamless integration.

[0073] It should be understood that although this specification includes some examples, none of these examples constitutes a single, independent technical solution. This descriptive style is merely for clarity. Those skilled in the art should consider this specification as a whole, and the technical solutions in the examples can be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0074] The detailed descriptions listed above are merely specific descriptions of feasible implementation methods of this application and are not intended to limit the scope of protection of this application. All equivalent implementation methods or modifications that do not depart from the teachings of this application should be included within the scope of protection of this application.

Claims

1. A heterogeneous data processing method based on machine learning, characterized in that, include: A dynamic knowledge graph for query reference is generated, which generates a graph structure based at least on first asset data from a first data source and second asset data from a second data source. The graph structure includes at least a first metadata vector representation in the first asset data, a second metadata vector representation in the second asset data, and an association index. The first metadata includes at least a first asset ontology and a first business tag, and the second metadata includes at least a second asset ontology and a second business tag. The association index establishes a connection based on the vector similarity between the first metadata and the second metadata. In response to the user's query request, the first metadata of the first data source and the second metadata of the second data source are queried according to the associated index. In response to the existence of duplicate data segments between the first asset data and the second asset data, the data source for returning the duplicate data segments is selected based on the communication service quality of the first data source and the second data source. The operation type, operation function and / or indexing capabilities of the first data source and the second data source are obtained, and the query request is decomposed to the first data source and the second data source according to the query request so that the first data source and the second data source return only the matching data. The first metadata and the second metadata are merged and processed according to the query requirements to meet the corresponding output format; The user's query request is natural language input. The user's query intent is determined based on the natural language input to generate an execution strategy that performs the query at least between the first data source and the second data source.

2. The heterogeneous data processing method based on machine learning according to claim 1, characterized in that, The method includes: Periodically scan at least one of the following: table structure, column definitions, data types, index information, statistical information, and data distribution of the first data source and the second data source; When the data difference exceeds a first threshold, an update to the dynamic knowledge graph is triggered.

3. The heterogeneous data processing method based on machine learning according to claim 1, characterized in that, The method includes: In response to an update to the first asset data, the first asset data is made to include the first asset data at a first time point and the first asset data at a second time point to construct context information.

4. The heterogeneous data processing method based on machine learning according to claim 1, characterized in that, The method includes: Obtain an asset procurement list for comparison with the asset ontology. The first asset ontology and the second asset ontology are extracted based on the asset procurement list using a pre-trained model.

5. The heterogeneous data processing method based on machine learning according to claim 1, characterized in that, The method includes: Analyze the user's historical query needs to predict the user's query preferences; Based on the query preferences, the returned data from the first data source and the second data source are obtained in advance and then merged and processed.

6. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the heterogeneous data processing method based on machine learning as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the heterogeneous data processing method based on machine learning as described in any one of claims 1-5.

Images