An intelligent search method and system based on multi-source heterogeneous data

Abstract

The application discloses an intelligent search method based on multi-source heterogeneous data, comprising the following steps: data introduction and preprocessing of multi-source heterogeneous data; data relationship extraction and data relationship representation of the preprocessed multi-source heterogeneous data to obtain multi-modal data; multi-modal correlation relationship mining and multi-modal semantic correlation fusion of the multi-modal data by using a dynamic heterogeneous graph attention embedding mechanism, and construction of a multi-modal semantic graph; cross-modal knowledge fusion of the multi-modal semantic graph by using a heterogeneous graph neural network, and construction of a heterogeneous cross-modal knowledge graph based on the cross-modal knowledge fusion result; subject analysis and element extraction processing of the search words of a target object, search of the target object by using the heterogeneous cross-modal knowledge graph to obtain multi-modal correlation information of the target object; holographic portrait of the target object according to the multi-modal correlation information of the target object, and output of an intelligent sorting result and the holographic portrait of the target object.

Description

Technical Field

[0001] This invention relates to the fields of multimodal knowledge graphs and computer vision, specifically to an intelligent search method and system based on multi-source heterogeneous data. Background Technology

[0002] In recent years, with the continuous development of artificial intelligence and big data technologies, and the rapid growth of massive heterogeneous data, especially in the field of public safety, information application systems are often highly coupled, have complex architectures, and are isolated from each other. This results in fragmented and isolated information resources with low levels of intelligence, hindering the organic synergy between data and business. Significant manual time is required for information retrieval and correlation analysis, and traditional manual processing methods are far from meeting the needs of current intelligent search. Utilizing artificial intelligence technologies such as knowledge graphs, natural language processing, and computer vision to achieve unified retrieval of heterogeneous data sources, improve spatiotemporal data correlation analysis capabilities, and enhance the intelligence of search engines can effectively improve retrieval speed and work efficiency.

[0003] Currently, retrieval tasks in the public safety field generally involve fuzzy attribute queries on target data using relational databases. The data processed is typically structured, making implementation relatively simple. Some business systems utilize Natural Language Processing (NLP) for semantic analysis of text, automatically extracting retrieval elements. However, the intelligence and accuracy of this automated retrieval are low, exhibiting problems such as slow search speeds, incomplete searches, and inaccurate results, failing to meet the actual needs of this field. For unstructured data such as images and audio, content-based retrieval relies on media content features and contextual semantics. Current multimedia search capabilities are limited, and their retrieval functions are incomplete, resulting in less than ideal performance in this area.

[0004] The existing intelligent search methods have the following technical defects: (1) Traditional search methods often have difficulty in mining potential relationships between isolated data from different sources, resulting in problems such as difficulty in integrating massive data and incomplete search, and cannot fully manage and apply massive data effectively; (2) Traditional keyword-based matching search is relatively simple, with many results of low relevance, slow retrieval speed and poor natural language understanding, and does not support the search of multi-source heterogeneous data; (3) For heterogeneous data such as images, audio and spatiotemporal geographic information, the existing search methods cannot provide a unified search service and cannot perform global search based on the local attributes and features of the target. Summary of the Invention

[0005] In view of the above problems, the present invention provides an intelligent search method and system based on multi-source heterogeneous data, in order to solve at least one of the above problems.

[0006] According to a first aspect of the present invention, an intelligent search method based on multi-source heterogeneous data is provided, comprising:

[0007] Based on the big data platform, multi-source heterogeneous data is introduced and preprocessed to obtain preprocessed multi-source heterogeneous data, which includes structured data, semi-structured data and unstructured data.

[0008] Data relationship extraction and representation are performed on structured data, semi-structured data, and unstructured data respectively to obtain multimodal data with data nodes and data node relationships. The multimodal data includes resource description framework triples, multi-group events, and concept-entity relationship graphs.

[0009] We utilize a dynamic heterogeneous graph attention embedding mechanism to mine multimodal relationships and fuse multimodal semantic associations in multimodal data, and construct a multimodal semantic graph based on the results of multimodal semantic association fusion.

[0010] Heterogeneous graph neural networks are used to perform cross-modal knowledge fusion on multimodal semantic graphs, and a heterogeneous cross-modal knowledge graph is constructed based on the results of cross-modal knowledge fusion.

[0011] The search terms for the target object are analyzed for topics and elements are extracted. Based on the processing results, a heterogeneous cross-modal knowledge graph is used to perform multi-path queries and searches on the target object to obtain multi-modal association information of the target object.

[0012] The system intelligently sorts the multimodal association information of the target object, creates a holographic profile of the target object based on the multimodal association information, and outputs the intelligent sorting results and the holographic profile of the target object.

[0013] According to embodiments of the present invention, the above-described data relationship extraction and data relationship representation of structured data, semi-structured data, and unstructured data to obtain multimodal data with data nodes and data node relationships includes:

[0014] ETL tools are used to extract and represent data relationships from structured data, resulting in a resource description framework triplet for the structured data.

[0015] Data relationships are extracted and represented from semi-structured data to obtain tuple events of semi-structured data;

[0016] Real-time speech recognition is performed on audio data in unstructured data to obtain speech recognition results. Entity recognition, data relationship extraction, and representation are then performed on the speech recognition results and file data in unstructured data to obtain text data processing results.

[0017] The target image detection model is used to identify image data in unstructured data to obtain image processing results. Based on the text data processing results and image processing results, a concept-entity relationship graph of unstructured data is obtained.

[0018] According to an embodiment of the present invention, the above-described image processing result obtained by recognizing image data in unstructured data using a target image detection model includes:

[0019] An image localization and detection network based on a target image detection model is used to locate and detect target images of different sizes, resulting in a target image database.

[0020] The target image recognition network of the target image detection model is used to extract features from unstructured image data, and the extracted features are then subjected to L2 normalization to obtain the embedded feature vector of the image data.

[0021] Calculate the Euclidean distance between the embedded feature vectors of the image data and each target image in the target image database;

[0022] Based on the Euclidean distance, the similarity between the embedded feature vector of the image data and each target image in the target image database is obtained, and the image data is identified based on the similarity to obtain the image processing result.

[0023] According to an embodiment of the present invention, the image localization and detection network described above includes a backbone network, a feature pyramid unit, a context modeling unit, and a multi-task learning unit.

[0024] According to embodiments of the present invention, the above-described method of using a dynamic heterogeneous graph attention embedding mechanism to mine multimodal relationships and fuse multimodal semantic relationships in multimodal data, and constructing a multimodal semantic graph based on the results of multimodal semantic relationship fusion, includes:

[0025] The node-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to measure the consistency of multimodal data with the same data node relationship, and the consistency measurement results are obtained.

[0026] We utilize the edge-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism to perform cross-modal complementarity measurement on multimodal data, and obtain the complementarity measurement results, which include the unique information of the single modality and the shared information of the multimodality.

[0027] The temporal attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to process the consistency measurement results and complementarity measurement results, complete the multimodal semantic association fusion of multimodal data, and construct a multimodal semantic graph based on the result of multimodal semantic association fusion.

[0028] According to an embodiment of the present invention, the above-described node-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to perform consistency measurement on multimodal data with the same data node relationship, and the resulting consistency measurement results include:

[0029] By using a node-level attention mechanism, neighborhood data nodes of multimodal data with the same data node relationship are aggregated to obtain the weights of the neighborhood data nodes of multimodal data with the same data node relationship.

[0030] Based on the weights of neighboring data nodes of multimodal data with the same data node relationship, data nodes are embedded in the multimodal data with the same data node relationship to complete the consistency measurement of multimodal data with the same data node relationship.

[0031] According to an embodiment of the present invention, the above-described method of using the edge-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism to perform cross-modal complementarity measurement on multimodal data, and obtaining the complementarity measurement result includes:

[0032] We utilize an edge-level attention mechanism to learn and normalize the different modalities of each data node in multimodal data, thereby obtaining the cross-modal weights of each data node.

[0033] Based on the cross-modal weights of each data node, cross-modal aggregation is performed on each data node in the multimodal data to complete the cross-modal complementarity measurement of the multimodal data.

[0034] According to an embodiment of the present invention, the above-described method of using a heterogeneous graph neural network to perform cross-modal knowledge fusion on a multimodal semantic graph and constructing a heterogeneous cross-modal knowledge graph based on the result of cross-modal knowledge fusion includes:

[0035] The encoding module of a heterogeneous graph neural network is used to aggregate and embed data nodes and data node relationships in a multimodal semantic graph.

[0036] Based on the aggregation and embedding results, the data nodes and data node relationships of the multimodal semantic graph are fused into a unified feature space;

[0037] The decoding module of a heterogeneous graph neural network is used to perform cross-modal decoding of data nodes and data node relationships in a multimodal semantic graph fused in a unified feature space, so as to obtain the result of cross-modal knowledge fusion, and construct a heterogeneous cross-modal knowledge graph based on the result of cross-modal knowledge fusion.

[0038] According to a second aspect of the present invention, an intelligent search system based on multi-source heterogeneous data is provided, comprising:

[0039] The data acquisition and preprocessing module is used to acquire and preprocess multi-source heterogeneous data based on the big data platform to obtain preprocessed multi-source heterogeneous data, which includes structured data, semi-structured data and unstructured data.

[0040] The multimodal data acquisition module is used to extract and represent data relationships from structured data, semi-structured data, and unstructured data respectively, to obtain multimodal data with data nodes and data node relationships. The multimodal data includes resource description framework triples, multi-group events, and concept-entity relationship graphs.

[0041] The multimodal semantic graph construction module is used to mine multimodal relationships and fuse multimodal semantic relationships in multimodal data using a dynamic heterogeneous graph attention embedding mechanism, and to construct a multimodal semantic graph based on the results of the multimodal semantic relationship fusion.

[0042] The heterogeneous cross-modal knowledge graph construction module is used to perform cross-modal knowledge fusion on multimodal semantic graphs using heterogeneous graph neural networks, and to construct a heterogeneous cross-modal knowledge graph based on the results of cross-modal knowledge fusion.

[0043] The search request processing module is used to perform topic parsing and element extraction on the search terms of the target object, and based on the processing results, to perform multi-path query and search on the target object using a heterogeneous cross-modal knowledge graph to obtain the multi-modal association information of the target object.

[0044] The intelligent sorting and result output module is used to intelligently sort the multimodal association information of the target object, create a holographic image of the target object based on the multimodal association information, and output the intelligent sorting result and the holographic image of the target object.

[0045] According to embodiments of the present invention, the search request processing module described above can perform comprehensive retrieval, batch retrieval, related retrieval, and image retrieval;

[0046] The comprehensive search includes full-text search, keyword combination search, range search, fuzzy search, pinyin and initial letter search, intelligent search suggestions, and advanced search.

[0047] The method of this invention can extract entities and relationships from multi-source heterogeneous data such as text, events, images, audio, and spatiotemporal data using a multivariate knowledge extraction framework. Through deep fusion of multi-source data, it constructs a holographic profile of the target, characterizing the target's attributes in all dimensions. It has the characteristics of being multimodal, cross-spatial, fully fused, and universally applicable. At the same time, the intelligent search method of this invention can realize the information association and fusion of multimodal heterogeneous data by utilizing the constructed knowledge graph, deeply mining the potential relationships between targets, making the search more intelligent. Attached Figure Description

[0048] Figure 1 This is a flowchart of an intelligent search method based on multi-source heterogeneous data according to an embodiment of the present invention;

[0049] Figure 2 This is a flowchart illustrating the acquisition of multimodal data according to an embodiment of the present invention;

[0050] Figure 3 This is a flowchart of obtaining image processing results according to an embodiment of the present invention;

[0051] Figure 4 This is a flowchart of constructing a multimodal semantic graph according to an embodiment of the present invention;

[0052] Figure 5 This is a flowchart of constructing a heterogeneous cross-modal knowledge graph according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of the intelligent search system based on multi-source heterogeneous data according to the present invention;

[0054] Figure 7 This is a schematic diagram of a process for an intelligent search method based on multi-source heterogeneous data according to another embodiment of the present invention;

[0055] Figure 8 This is a flowchart of facial target feature comparison according to another embodiment of the present invention;

[0056] Figure 9 This is a functional structure diagram of an intelligent search system based on multi-source heterogeneous data according to another embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0058] Utilizing artificial intelligence technologies such as knowledge graphs, natural language processing, and computer vision to achieve unified retrieval of heterogeneous data sources can improve the ability of spatiotemporal data correlation analysis and the intelligence level of search engines, effectively increasing retrieval speed and work efficiency. However, existing intelligent search methods suffer from problems such as poor processing capabilities for massive multimodal data.

[0059] Therefore, this invention provides an intelligent search method and system based on multi-source heterogeneous data, which can manage a large amount of data accumulated over a long period of time, fully and comprehensively explore the potential relationships between targets by constructing a knowledge graph, and reason and complete unknown information, thereby solving problems such as low search intelligence, scattered and isolated data, and single search method.

[0060] It should be particularly noted that the image data (such as facial image data) involved in this invention strictly comply with the provisions of relevant laws and regulations and meet the requirements of relevant public order and good morals during the acquisition, processing, application and storage processes.

[0061] Figure 1 This is a flowchart of an intelligent search method based on multi-source heterogeneous data according to an embodiment of the present invention.

[0062] like Figure 1 As shown, the above-mentioned intelligent search method based on multi-source heterogeneous data includes operations S110 to S160.

[0063] In operation S110, multi-source heterogeneous data is introduced and preprocessed based on the big data platform to obtain preprocessed multi-source heterogeneous data, which includes structured data, semi-structured data and unstructured data.

[0064] The aforementioned multi-source heterogeneous data consists of massive amounts of data in the hundreds of millions, and therefore can be integrated and preprocessed through a big data platform.

[0065] In operation S120, data relationship extraction and data relationship representation are performed on structured data, semi-structured data and unstructured data respectively, to obtain multimodal data with data nodes and data node relationships. The multimodal data includes resource description framework triples, multi-group events and concept-entity relationship graphs.

[0066] For structured data accessed from a unified data source, ETL tools are used to extract data and construct RDF triples (Resource Description Framework (RDF) is a data model that uses XML syntax to describe the characteristics of Web resources and the relationships between resources). For semi-structured data, tuple events are constructed through relevant data processing operations (such as using wrappers), thus completing the process of extracting and representing knowledge from both types of data.

[0067] The aforementioned ETL tool is Extract-Transform-Load, which describes the process of extracting, transforming, and loading data from the source to the destination. It is a type of data warehouse technology.

[0068] For unstructured data such as text, audio, and images, entity recognition, relation extraction, real-time speech recognition, and object detection algorithms are used to obtain concepts, entities, and their relationships, providing data support for the construction of multimodal semantic graphs.

[0069] In operation S130, a dynamic heterogeneous graph attention embedding mechanism is used to mine multimodal relationships and fuse multimodal semantic associations in multimodal data, and a multimodal semantic graph is constructed based on the results of multimodal semantic association fusion.

[0070] By mining the correlations at different conceptual levels in multimodal data, establishing cross-modal associations between nodes, constructing a multimodal semantic graph, and intelligently associating multi-source heterogeneous data and aligning target multimodal attributes, the association and fusion of multimodal semantics can be completed.

[0071] In operation S140, a heterogeneous graph neural network is used to perform cross-modal knowledge fusion on the multimodal semantic graph, and a heterogeneous cross-modal knowledge graph is constructed based on the results of cross-modal knowledge fusion.

[0072] By using heterogeneous graph neural networks to establish multiple relationships between entity objects, a heterogeneous cross-modal large-scale knowledge graph is constructed, thus completing the knowledge fusion process.

[0073] In operation S150, the search terms for the target object are parsed and elements are extracted. Based on the processing results, a multi-path query and search are performed on the target object using a heterogeneous cross-modal knowledge graph to obtain the multi-modal association information of the target object.

[0074] By performing topic analysis and element extraction on the search terms, multi-path queries and explorations are conducted on the constructed knowledge graph to target the search.

[0075] In operation S160, the multimodal association information of the target object is intelligently sorted, a holographic image of the target object is created based on the multimodal association information of the target object, and the intelligent sorting result and the holographic image of the target object are output.

[0076] The method of this invention can extract entities and relationships from multi-source heterogeneous data such as text, events, images, audio, and spatiotemporal data using a multivariate knowledge extraction framework. Through deep fusion of multi-source data, it constructs a holographic profile of the target, comprehensively depicting the target's attributes and possessing multimodal, cross-spatial, fully fused, and general object characteristics. Simultaneously, the intelligent search method of this invention utilizes a constructed knowledge graph to achieve information association and fusion of multimodal heterogeneous data, deeply mining potential relationships between targets, making the search more intelligent. This can effectively solve practical problems in the field of public safety, realizing intelligent search, analysis, and judgment through human-machine collaboration. Furthermore, unlike matching retrieval between homogeneous data, the method of this invention can establish relationships and perform feature analysis and matching between cross-media data. Based on the semantics of search terms, it enables interactive retrieval between heterogeneous media data, possessing the ability to organize and rapidly retrieve massive amounts of multi-source data. It also provides a unified service platform for general, basic data resource query and retrieval functions and common algorithms.

[0077] Figure 2 This is a flowchart of acquiring multimodal data according to an embodiment of the present invention.

[0078] like Figure 2 As shown, the above operations S210 to S240 respectively extract and represent data relationships from structured data, semi-structured data, and unstructured data to obtain multimodal data with data nodes and data node relationships.

[0079] In operation S210, ETL tools are used to extract and represent data relationships from structured data, resulting in resource description framework triples for the structured data.

[0080] In operation S220, data relationships are extracted and represented from the semi-structured data to obtain tuple events of the semi-structured data.

[0081] In operation S230, real-time speech recognition is performed on the audio data in the unstructured data to obtain the speech recognition result. Then, entity recognition, data relationship extraction and representation are performed on the speech recognition result and the file data in the unstructured data to obtain the text data processing result.

[0082] In operation S240, the target image detection model is used to identify image data in unstructured data to obtain image processing results, and a concept-entity relationship graph of unstructured data is obtained based on the text data processing results and image processing results.

[0083] Figure 3 This is a flowchart of obtaining image processing results according to an embodiment of the present invention.

[0084] like Figure 3 As shown, the above-mentioned image processing results obtained by using the target image detection model to identify image data in unstructured data include operations S310 to S340.

[0085] In operation S310, the image localization and detection network of the target image detection model is used to locate and detect target images of different sizes, thereby obtaining a target image database.

[0086] In operation S320, the target image recognition network of the target image detection model is used to extract features from the image data in the unstructured data, and the extracted features are then subjected to L2 normalization to obtain the embedded feature vector of the image data.

[0087] In operation S330, the Euclidean distance between the embedded feature vector of the image data and each target image in the target image database is calculated.

[0088] In operation S340, the similarity between the embedded feature vector of the image data and each target image in the target image database is obtained based on the Euclidean distance. The image data is then identified based on the similarity to obtain the image processing result.

[0089] According to an embodiment of the present invention, the image localization and detection network described above includes a backbone network, a feature pyramid unit, a context modeling unit, and a multi-task learning unit.

[0090] The following describes specific embodiments. Figure 3 The process shown will be explained in further detail.

[0091] This invention utilizes a target detection model primarily targeting human faces, supporting the facial search function in intelligent search systems. It employs an improved RetinaFace network (target image localization and detection network) to detect and locate faces, providing accurate location information for faces at all scales to capture faces and construct a face database. Then, a FaceNet network (target image recognition network) is used to extract features from faces in real-time images using a deep CNN. After L2 normalization, the embedding vector representing the face's features is obtained. The face features are compared with the face database, and the Euclidean distance for each face is calculated for similarity evaluation. The improved RetinaFace network employs a multi-task learning strategy, detecting and predicting face positions. The face scoring algorithm also regresses the location information of five facial key points. The network structure consists of four components: a backbone network, an FPN feature pyramid, context modeling, and multi-task learning. The backbone network uses a ResNet50 network for feature extraction. To enhance the network's contextual reasoning ability for small target faces, a context module is added to the feature pyramid. This aims to enhance the receptive field obtained from the Euclidean distance grid, enabling pixel-level localization of faces of different sizes. In the multi-task learning branch, since the dense 3D locations corresponding to each face pixel contribute relatively little to the multi-task loss function, and face occlusion can severely affect recognition accuracy, the dense point regression loss is replaced with an occlusion loss. The improved multi-task loss function... Formula (1) represents:

[0092] (1),

[0093] In formula (1), For face classification loss, To predict the probability of it being a face, The true probability is 0 or 1. The Softmax function is used for binary classification, which distinguishes between face and background information. Regression loss for face outlines, and They represent The positions of the predicted bounding boxes and the ground truth bounding boxes were calculated using the Smooth L1 robust regression function. The weight for the bounding box regression loss is set to 0.25; For facial landmark regression functions, and These represent the predicted and actual coordinates of five key points corresponding to a positive sample face, respectively, and have been normalized based on the coordinates of the anchor point center. The weights for the facial landmark regression loss are set to 0.1; For the occlusion loss function, and These represent the predicted probability of occlusion and the true label of the occlusion, respectively, corresponding to the positive sample face. Based on the degree of occlusion, they are categorized as no occlusion, partial occlusion, moderate occlusion, and severe occlusion. Possible values ​​are: The weight for the occlusion loss is set to 1.

[0094] Figure 4 This is a flowchart of constructing a multimodal semantic graph according to an embodiment of the present invention.

[0095] like Figure 4 As shown, the above-mentioned use of dynamic heterogeneous graph attention embedding mechanism to mine multimodal relationships and fuse multimodal semantic relationships in multimodal data, and to construct a multimodal semantic graph based on the results of multimodal semantic relationship fusion includes operations S410 to S430.

[0096] When operating S410, the node-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to perform consistency measurement on multimodal data with the same data node relationship, and the consistency measurement result is obtained.

[0097] In operation S420, the edge-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to perform cross-modal complementarity measurement on multimodal data, and the complementarity measurement results are obtained. The complementarity measurement results include the unique information of the single modality and the shared information of the multimodality.

[0098] In the S430 operation, the temporal attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to process the consistency measurement results and complementarity measurement results, complete the multimodal semantic association fusion of multimodal data, and construct a multimodal semantic graph based on the result of multimodal semantic association fusion.

[0099] According to an embodiment of the present invention, the above-described node-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to perform consistency measurement on multimodal data with the same data node relationship, and the resulting consistency measurement results include:

[0100] By using a node-level attention mechanism, neighborhood data nodes of multimodal data with the same data node relationship are aggregated to obtain the weights of the neighborhood data nodes of multimodal data with the same data node relationship.

[0101] Based on the weights of neighboring data nodes of multimodal data with the same data node relationship, data nodes are embedded in the multimodal data with the same data node relationship to complete the consistency measurement of multimodal data with the same data node relationship.

[0102] According to an embodiment of the present invention, the above-described method of using the edge-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism to perform cross-modal complementarity measurement on multimodal data, and obtaining the complementarity measurement result includes:

[0103] We utilize an edge-level attention mechanism to learn and normalize the different modalities of each data node in multimodal data, thereby obtaining the cross-modal weights of each data node.

[0104] Based on the cross-modal weights of each data node, cross-modal aggregation is performed on each data node in the multimodal data to complete the cross-modal complementarity measurement of the multimodal data.

[0105] The process of constructing a multimodal semantic graph will be explained in further detail below with reference to specific implementation methods.

[0106] This invention utilizes a dynamic heterogeneous graph embedding method to construct a multimodal semantic graph, thereby achieving multimodal semantic association and fusion. It employs different levels of attention to learn subgraph embeddings at different levels, capturing the association strength of aggregated semantics across different modalities. The dynamic heterogeneous graph attention embedding mechanism consists of three parts: node-level attention, edge-level attention, and temporal attention, each using a different attention layer to aggregate different subgraph information. Node-level attention enables consistency measurement between different modalities of subgraphs with the same edge type; edge-level attention measures cross-modal complementarity between different subgraphs, preserving unique information of single modalities and shared information across multiple modalities; and temporal attention provides a general representation of the multimodal graph, enhancing the ability to understand natural language and enabling interactive intelligent retrieval. The node-level attention layer aims to learn the neighborhood weights of each node and complete subgraph embedding by aggregating features of these important neighborhoods. For each time step snapshot, it is divided into different subgraphs according to edge type, and a self-attention mechanism is used to embed nodes in each subgraph with the same edge type. For edge types... and the A snapshot, data node Weighting coefficients It can be expressed by formula (2):

[0107] (2),

[0108] In formula (2), It is the activation function, specifically the ReLU function. It is a data node The initial feature vector, Indicates edge type The linear transformation projection matrix, Indicates the join operation. Indicates snapshot Middle edge class Data Node Sampling neighbor nodes, Represents edge class The parameterized weight vector of the attention function, through the calculated weight coefficients, aggregates the latent embeddings of neighbors, thus obtaining the edge class of data node X. and the The final result at each snapshot is shown in formula (3):

[0109] (3),

[0110] In formula (3), It is an edge type and the Data nodes of a snapshot To obtain stable and effective features, the clustered embedding employs a node-level attention model with a multi-head mechanism; specifically, it runs in parallel. Each node has an independent node-level attention layer, and these learned features are concatenated as the output embedding. While node-level attention layers can capture information specific to a single edge type, heterogeneous multimodal data typically contains multiple edge types. To integrate multiple edge-specific information from each node, edge-level attention layers are used to learn the importance weights for different edge types. The importance of each edge type is calculated by a multilayer perceptron (MLP) layer. These different types of edge-specific information are aggregated to generate new embeddings. and the Data nodes of a snapshot The normalized weighting coefficients can be expressed by formula (4):

[0111] (4),

[0112] In formula (4) This is represented as an attention vector at the edge level. and These are the parameters of a single-layer MLP, namely the learnable weight matrix and the bias vector. These two parameters are shared across different time snapshots and different edge types. This represents mapping a specific edge embedding into the same feature space obtained by inputting it into a nonlinear transformation function, and calculating the mapped specific edge embedding and the edge-level attention parameterization vector. The similarity between them is used to measure the importance coefficient of the input specific edge embedding, and then these specific edge embeddings are aggregated to generate the first... Data nodes of a snapshot The final representation features This can be expressed by formula (5):

[0113] (5),

[0114] In formula (5), Represents data nodes Integrating node-level attention Embedded, In a vector space, after obtaining the node embeddings for each temporal snapshot, the node embeddings for temporal attention are calculated by aggregating the node embeddings within a series of temporal snapshots. To process all historical times of a node, represented as Time-level attention can capture the characteristics of temporal evolution.

[0115] Figure 5 This is a flowchart of constructing a heterogeneous cross-modal knowledge graph according to an embodiment of the present invention.

[0116] like Figure 5 As shown, the above-mentioned use of heterogeneous graph neural networks to perform cross-modal knowledge fusion on multimodal semantic graphs and to construct a heterogeneous cross-modal knowledge graph based on the results of cross-modal knowledge fusion includes operations S510 to S530.

[0117] When operating the S510, the encoding module of the heterogeneous graph neural network is used to aggregate and embed data nodes and data node relationships in the multimodal semantic graph.

[0118] In the S520 operation, based on the aggregation embedding results, the data nodes and data node relationships of the multimodal semantic graph are fused into a unified feature space.

[0119] When operating the S530, the decoding module of the heterogeneous graph neural network is used to perform cross-modal decoding on the data nodes and data node relationships of the multimodal semantic graph fused in a unified feature space, so as to obtain the result of cross-modal knowledge fusion, and construct a heterogeneous cross-modal knowledge graph based on the result of cross-modal knowledge fusion.

[0120] The flowchart for constructing a heterogeneous cross-modal knowledge graph will be further explained in detail below with reference to specific embodiments.

[0121] The heterogeneous graph neural network used in this invention can jointly learn node embeddings and relation representations in a multi-relation graph. An encoder-decoder module is added to the graph neural network. During the encoding stage, node embeddings and relation embeddings are aggregated, fusing the initial representations of nodes and relations into a unified feature space. Then, during the decoding stage, the TransE method is used to decode the triples. Specifically, the edges in the multi-relation graph can be represented as... This indicates the existence of a path from the data node. Pointing to data nodes The type is There exists an edge with a corresponding reverse edge. And self-connecting edges that can be connected to themselves Considering the above three edge types on the multi-relation graph, the process of aggregating the corresponding neighbors in the multi-relation graph neural network is shown in formula (6):

[0122] (6),

[0123] In formula (6), It is a data node In multiple relationships The following is a collection of neighbors. These are the three edge types mentioned above. express The projection matrices corresponding to the three edge types at time t. and Representing data nodes and Feature embedding representation, The function takes into account the influence of node and edge relationships.

[0124] Based on the unified representation and management of multimodal, heterogeneous, and dynamic target information data, after constructing object-oriented information association and fusion, it is necessary to perform unified semantic expression modeling for different types of information to achieve a unified expression of the connotation of multimodal information. Heterogeneous graph neural networks are used for knowledge fusion, embedding non-Euclidean information in virtual and real spaces into Euclidean space for unified representation, further achieving feature and semantic level fusion. By aligning multimodal information at the same granularity and semantics through modeling, the construction of a cross-modal knowledge graph is completed. Based on this, combined with natural language processing and other technologies, functions such as intelligent semantic parsing of search queries, intelligent recommendation of search terms, and deep association retrieval are realized.

[0125] The topic parsing and element extraction used in this invention are achieved by using a slot filling algorithm to parse the input search terms into topics and assign them to the corresponding predefined intelligent search engines. Based on this, a named entity recognition algorithm is used to extract elements from the question, extracting triples of entity attribute values ​​from the text. Entities include elements such as people, organizations, place names, vehicles, and contact information. Finally, combined with a rule matching method, corresponding path queries are performed on the constructed knowledge graph around the identified main entities.

[0126] Figure 6 This is a schematic diagram of the structure of an intelligent search system based on multi-source heterogeneous data according to the present invention.

[0127] like Figure 6 As shown, the intelligent search system based on multi-source heterogeneous data includes a data acquisition and preprocessing module 610, a multimodal data acquisition module 620, a multimodal semantic graph construction module 630, a heterogeneous cross-modal knowledge graph construction module 640, a search request processing module 650, and an intelligent sorting and result output module 660.

[0128] The data acquisition and preprocessing module 610 is used to acquire and preprocess multi-source heterogeneous data based on a big data platform to obtain preprocessed multi-source heterogeneous data, which includes structured data, semi-structured data and unstructured data.

[0129] The multimodal data acquisition module 620 is used to extract and represent data relationships from structured data, semi-structured data and unstructured data respectively, to obtain multimodal data with data nodes and data node relationships. The multimodal data includes resource description framework triples, multi-group events and concept-entity relationship graphs.

[0130] The multimodal semantic graph construction module 630 is used to mine multimodal relationships and fuse multimodal semantic relationships in multimodal data using a dynamic heterogeneous graph attention embedding mechanism, and to construct a multimodal semantic graph based on the results of the multimodal semantic relationship fusion.

[0131] The heterogeneous cross-modal knowledge graph construction module 640 is used to perform cross-modal knowledge fusion on multimodal semantic graphs using heterogeneous graph neural networks, and to construct a heterogeneous cross-modal knowledge graph based on the results of cross-modal knowledge fusion.

[0132] The search request processing module 650 is used to perform topic parsing and element extraction on the search terms of the target object, and based on the processing results, to perform multi-path query and search on the target object using a heterogeneous cross-modal knowledge graph to obtain the multi-modal association information of the target object.

[0133] The intelligent sorting and result output module 660 is used to intelligently sort the multimodal association information of the target object, create a holographic image of the target object based on the multimodal association information of the target object, and output the intelligent sorting result and the holographic image of the target object.

[0134] According to embodiments of the present invention, the search request processing module described above can perform comprehensive retrieval, batch retrieval, related retrieval, and image retrieval;

[0135] The comprehensive search includes full-text search, keyword combination search, range search, fuzzy search, pinyin and initial letter search, intelligent search suggestions, and advanced search.

[0136] The aforementioned intelligent search system based on multi-source heterogeneous data includes comprehensive search, batch search, related search, and facial recognition search. Comprehensive search integrates multiple functions within a single input box, such as full-text search, keyword combination search, range search, fuzzy search, pinyin and initial letter search, intelligent search suggestions, and advanced search. Search targets cover multiple entities including people, vehicles, organizations, events, and contact information. Batch search includes file search and ultra-large file search. File search performs batch retrieval by parsing file content, while ultra-large file search performs large-scale retrieval tasks through offline batch processing. Related search includes single-relationship search and multi-relationship combined joint search, supporting the search of relationships between multiple entities and shortest path search. Search results are displayed in the form of a knowledge graph. Facial recognition search supports uploading facial images for searching, using face detection and recognition algorithms to calculate and match similarity against targets in a massive face database for accurate searching. This system can achieve information fusion and association of heterogeneous data, utilizing a constructed multimodal knowledge graph to mine potential relationships between targets, making the search more intelligent.

[0137] To better illustrate the advantages of the intelligent search method for multi-source heterogeneous data provided by the present invention, the following describes another embodiment and... Figures 7-9 The present invention will be further described below.

[0138] Figure 7 This is a schematic diagram of a process for an intelligent search method based on multi-source heterogeneous data according to another embodiment of the present invention.

[0139] Figure 8 This is a flowchart of facial target feature comparison according to another embodiment of the present invention.

[0140] Figure 9 This is a functional structure diagram of an intelligent search system based on multi-source heterogeneous data according to another embodiment of the present invention.

[0141] like Figure 7As shown, the system first performs massive data ingestion and preprocessing. Based on the Datax (a heterogeneous data source exchange tool) offline data processing tool, a distributed data synchronization system capable of handling tens of thousands of records per second is implemented. Resource scheduling reduces the pressure on individual machines and improves overall system performance. Furthermore, it adapts to diverse heterogeneous databases, enabling batch synchronization of databases such as Oracle, MySQL, and Hive. By implementing a proprietary structured query language toolkit, tasks such as exploring and reconciling massive amounts of data are accurately executed.

[0142] In the multimodal data processing stage, structured data is processed using the distributed computing framework Spark. Full operators cover structured, semi-structured, and unstructured data sources, while computational operators cover data deduplication, filtering, association extraction, and association backfilling. Massive data computation is achieved through visual configuration, and the results are stored in different data storage devices. For unstructured data such as text, audio, and images, the BERT model is used for entity recognition and relation extraction in text data, and a real-time speech recognition algorithm is used to process audio data. For facial image data, this invention uses an improved RetinaFace network for target detection and localization. The process includes building a face database, face cropping and alignment, face angle correction and encoding, face feature comparison, and outputting the face recognition results. When constructing a face database, it is necessary to traverse all images in the database, use the Retinaface network to detect the face position in each image, crop the detected face portion, and perform face correction and alignment. During alignment, it is necessary to calculate the tilt angle of the eye line relative to the horizontal line and the coordinates of the image center, using binocular coordinates for rotational alignment. Then, the FaceNet network is used to encode the faces, and the results of all face encoding are saved as .npy files. The flowchart for face target feature comparison is as follows. Figure 8 As shown, each facial feature in the real-time image is obtained through the Facenet network. The distance between each facial feature and all targets in the face database is calculated, and a similarity comparison is performed to obtain the sequence number of the most similar face. Finally, a distance threshold is used for discrimination. The distance threshold is set to 0.7. If it is less than the distance threshold, the face corresponding to this sequence number is judged as the recognition result.

[0143] In the multimodal semantic graph construction stage, dynamic heterogeneous graph embedding utilizes different levels of attention to learn subgraph embeddings at different levels. In its implementation, considering the usability and efficiency of the algorithm, a multi-head attention mechanism-based model layer fuses multiple dimensions through multiple heads, and each head can generate a different attention distribution, addressing long-term dependencies in the information. First, a bidirectional gated recurrent unit (Bi-GRU) is used to consider the inter-modal semantic dependencies and obtain the internal information of each modality. Second, a cross-modal attention interaction network layer combines the internal information of each modality with the interactions between modalities. Finally, an attention mechanism based on the contribution level of each modality is introduced to determine the attention weights of each modality and effectively align the features of each modality.

[0144] In the cross-modal knowledge association and fusion stage, entities and relations in a heterogeneous graph neural network can be represented as two complementary subgraphs, with nodes iteratively updated based on the values ​​of their surrounding edges. During the encoding stage, node embeddings and relation embeddings are aggregated using variations of four different node aggregation techniques: average pooling, max pooling, and LSTM. This maps entities and relations from different heterogeneous graphs to a unified vector space, transforming the mapping of entities and relations into a vector distance calculation problem. In the decoding stage, TransE, based on the distributed vector representation of entities and relations, treats the relations in each triplet instance as the sum of vectors from the head entity to the tail entity. Its goal is to maximize the distance between the closest positive and negative samples. Negative samples are constructed by replacing a self-constructed error; either by randomly replacing the head entity to create an incorrect triplet, or by randomly replacing the tail entity to form an incorrect triplet. The process of aggregating the corresponding neighbors of multi-relationship nodes using a heterogeneous graph neural network starts by sampling neighbor nodes from a fixed point, and at the same time, there is a certain probability of returning to the fixed point to start sampling again, so as to ensure that all types are sampled until a fixed number is reached. Then, these sampled nodes are classified according to type, and the nodes with the top k frequency of occurrence in the same type are selected as neighbor nodes.

[0145] During intelligent search, entity elements are provided for intelligent search problem identification through keyword topic parsing and element extraction. Triple relationships include information such as person-age, person-address, person-occupation, person-date of birth, person-registered residence, person-place of birth, person-gender, person-religious belief, person-ethnicity, person-education level, person-marital status, and person-height. This invention also provides an intelligent search system based on multi-source heterogeneous data, such as... Figure 9The diagram shows the functional structure of the intelligent search system based on multi-source heterogeneous data in the invention. The system includes: comprehensive search, batch search, related search, and facial recognition search. The comprehensive search integrates multiple functions within a single input box, such as full-text search, keyword combination search, range search, fuzzy search, pinyin and initial letter search, intelligent search suggestions, and advanced search. Search targets cover multiple entities including people, vehicles, organizations, events, and contact information. Batch search includes file search and ultra-large file search. File search performs batch retrieval by parsing file content, while ultra-large file search performs large-scale retrieval tasks through offline batch processing. Related search includes single-relationship search and multi-relationship combined joint search, supporting the search of relationships between multiple entities and shortest path search. Search results are displayed in the form of a knowledge graph. Facial recognition search supports uploading facial images for searching. Through facial detection and recognition algorithms, it calculates and matches similarity against targets in a massive facial database, thereby achieving accurate searching. The system can achieve information fusion and association of heterogeneous data, utilizing a constructed multimodal knowledge graph to mine potential relationships between targets, making the search more intelligent. By analyzing and designing the system functions, a complete solution and framework were established to address the problem of accurate search of massive, multi-source, heterogeneous data in a big data environment.

[0146] The final output includes the coordinates of the bounding box of the target in the image and the corresponding target category.

[0147] The intelligent search method and system based on multi-source heterogeneous data of the present invention can explore the potential correlation between targets, establish relationships between cross-modal data, and perform feature analysis and matching. It can realize interactive retrieval between heterogeneous modal data based on the semantics of search terms, and has the ability to organize and associate multi-source massive data and perform rapid retrieval. It solves the problems of low search intelligence, scattered and isolated data, and single search method, and can be applied to multiple fields.

[0148] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent search method based on multi-source heterogeneous data, comprising: The multi-source heterogeneous data is accessed and preprocessed based on a big data platform to obtain preprocessed multi-source heterogeneous data, which includes structured data, semi-structured data and unstructured data. Data relationship extraction and representation are performed on the structured data, semi-structured data, and unstructured data respectively to obtain multimodal data with data nodes and data node relationships. The multimodal data includes resource description framework triples, multi-group events, and concept-entity relationship graphs. The dynamic heterogeneous graph attention embedding mechanism is used to mine multimodal relationships and fuse multimodal semantic relationships in the multimodal data, and a multimodal semantic graph is constructed based on the results of the multimodal semantic relationship fusion. A heterogeneous graph neural network is used to perform cross-modal knowledge fusion on the multimodal semantic graph, and a heterogeneous cross-modal knowledge graph is constructed based on the results of the cross-modal knowledge fusion. The search terms for the target object are subjected to topic parsing and element extraction. Based on the processing results, the heterogeneous cross-modal knowledge graph is used to perform multi-path query and search on the target object to obtain the multi-modal association information of the target object. The multimodal association information of the target object is intelligently sorted, a holographic image of the target object is generated based on the multimodal association information of the target object, and the intelligent sorting result and the holographic image of the target object are output.

2. The method according to claim 1, wherein, The step of extracting and representing data relationships from the structured data, semi-structured data, and unstructured data to obtain multimodal data with data nodes and data node relationships includes: The structured data is extracted and represented using ETL tools to obtain the resource description framework triples of the structured data; Data relationships are extracted and represented from the semi-structured data to obtain tuple events of the semi-structured data; Real-time speech recognition is performed on the audio data in the unstructured data to obtain speech recognition results. Entity recognition and data relationship extraction and representation are then performed on the speech recognition results and the file data in the unstructured data to obtain text data processing results. The image data in the unstructured data is identified using a target image detection model to obtain image processing results. Based on the text data processing results and the image processing results, a concept-entity relationship graph of the unstructured data is obtained.

3. The method according to claim 2, wherein, The image processing results obtained by using a target image detection model to identify image data in the unstructured data include: The image localization and detection network of the target image detection model is used to locate and detect target images of different sizes to obtain a target image database; The target image recognition network of the target image detection model is used to extract features from the image data in the unstructured data, and the extracted features are then subjected to L2 normalization to obtain the embedded feature vector of the image data. Calculate the Euclidean distance between the embedded feature vector of the image data and each target image in the target image database; Based on the Euclidean distance, the similarity between the embedded feature vector of the image data and each target image in the target image database is obtained, and the image data is identified based on the similarity to obtain the image processing result.

4. The method according to claim 3, wherein, The image localization and detection network includes a backbone network, a feature pyramid unit, a context modeling unit, and a multi-task learning unit.

5. The method according to claim 1, wherein, The step of using a dynamic heterogeneous graph attention embedding mechanism to mine multimodal relationships and fuse multimodal semantic associations in the multimodal data, and constructing a multimodal semantic graph based on the results of the multimodal semantic association fusion, includes: The node-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to measure the consistency of multimodal data with the same data node relationship, and the consistency measurement results are obtained. The multimodal data is subjected to cross-modal complementarity measurement using the edge-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism, and the complementarity measurement result is obtained. The complementarity measurement result includes the unique information of the single modality and the shared information of the multimodality. The consistency measurement result and the complementarity measurement result are processed by the temporal attention mechanism in the dynamic heterogeneous graph attention embedding mechanism to complete the multimodal semantic association fusion of multimodal data, and a multimodal semantic graph is constructed based on the result of the multimodal semantic association fusion.

6. The method according to claim 5, wherein, The node-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism is used to perform consistency measurement on multimodal data with the same data node relationship, and the consistency measurement results include: The node-level attention mechanism is used to aggregate neighborhood data nodes of the multimodal data with the same data node relationship, so as to obtain the weight of the neighborhood data nodes of the multimodal data with the same data node relationship. Based on the weights of the neighboring data nodes of the multimodal data with the same data node relationship, the data nodes of the multimodal data with the same data node relationship are embedded to complete the consistency measurement of the multimodal data with the same data node relationship.

7. The method according to claim 5, wherein, The method of using the edge-level attention mechanism in the dynamic heterogeneous graph attention embedding mechanism to perform cross-modal complementarity measurement on the multimodal data yields the following complementarity measurement results: The edge-level attention mechanism is used to learn and normalize the different modalities of each data node in the multimodal data to obtain the cross-modal weights of each data node. Based on the cross-modal weights of each data node, cross-modal aggregation is performed on each data node in the multimodal data to complete the cross-modal complementarity measurement of the multimodal data.

8. The method according to claim 1, wherein, The step of using a heterogeneous graph neural network to perform cross-modal knowledge fusion on the multimodal semantic graph, and constructing a heterogeneous cross-modal knowledge graph based on the results of cross-modal knowledge fusion, includes: The encoding module of the heterogeneous graph neural network is used to aggregate and embed data nodes and data node relationships in the multimodal semantic graph; Based on the aggregation and embedding results, the data nodes and data node relationships of the multimodal semantic graph are fused into a unified feature space; The decoding module of the heterogeneous graph neural network is used to perform cross-modal decoding on the data nodes and data node relationships of the multimodal semantic graph fused in a unified feature space to obtain the result of cross-modal knowledge fusion, and a heterogeneous cross-modal knowledge graph is constructed based on the result of cross-modal knowledge fusion.

9. An intelligent search system based on multi-source heterogeneous data, comprising: The data acquisition and preprocessing module is used to acquire and preprocess the multi-source heterogeneous data based on the big data platform to obtain preprocessed multi-source heterogeneous data, wherein the preprocessed multi-source heterogeneous data includes structured data, semi-structured data and unstructured data. The multimodal data acquisition module is used to extract and represent data relationships from the structured data, the semi-structured data, and the unstructured data respectively, to obtain multimodal data with data nodes and data node relationships. The multimodal data includes resource description framework triples, multi-group events, and concept-entity relationship graphs. The multimodal semantic graph construction module is used to mine multimodal relationships and fuse multimodal semantic relationships in the multimodal data using a dynamic heterogeneous graph attention embedding mechanism, and to construct a multimodal semantic graph based on the results of the multimodal semantic relationship fusion. The heterogeneous cross-modal knowledge graph construction module is used to perform cross-modal knowledge fusion on the multimodal semantic graph using a heterogeneous graph neural network, and to construct a heterogeneous cross-modal knowledge graph based on the result of cross-modal knowledge fusion. The search request processing module is used to perform topic parsing and element extraction on the search terms of the target object, and based on the processing results, to perform multi-path query and search on the target object using the heterogeneous cross-modal knowledge graph to obtain the multi-modal association information of the target object; The intelligent sorting and result output module is used to intelligently sort the multimodal association information of the target object, create a holographic image of the target object based on the multimodal association information of the target object, and output the intelligent sorting result and the holographic image of the target object.

10. The system according to claim 9, wherein, The search request processing module can perform comprehensive search, batch search, related search, and image search. The comprehensive search includes full-text search, keyword combination search, range search, fuzzy search, pinyin and initial letter search, intelligent search suggestions, and advanced search.

Images