Multi-modal data processing method and system based on knowledge graph

Abstract

The embodiment of the invention provides a multi-modal data processing method and system based on a knowledge graph, and aims to solve the problems of semantic association deficiency and potential association mining difficulty in multi-source information processing. The method comprises the following steps: firstly, acquiring a multi-source information set containing different expression form information units and source identifiers, and performing entity mapping on the multi-source information set and a preset semantic association network to generate an information unit set with semantic association tags; performing cross-source feature association processing to obtain a comprehensive feature set; then, a semantic reasoning rule is called for association extension, and an extension feature set is generated; and finally, based on the extended feature set, generating an information processing result conforming to an application scene and feeding back the information processing result to a corresponding interface, thereby effectively integrating multi-source information and mining potential association.

Description

Technical Field

[0001] This application relates to the field of knowledge graph technology, and more specifically, to a multimodal data processing method and system based on knowledge graphs. Background Technology

[0002] In today's era of digital information explosion, multi-source information is widely present in various fields, such as healthcare, finance, and scientific research. This multi-source information has a rich variety of forms of expression, including text, images, audio, and video, and comes from a wide range of sources, such as different databases, sensors, and social media platforms.

[0003] Currently, the processing of multi-source information faces the following main challenges. On the one hand, existing information processing methods often handle information from different sources and in different forms in isolation, lacking effective means to semantically link and integrate this information. For example, when processing medical information, patient medical records, medical images, and examination reports come from different systems, making it difficult to establish organic connections between these pieces of information. This hinders doctors from comprehensively and accurately obtaining patient information during diagnosis and treatment. On the other hand, traditional information processing technologies struggle to uncover potential relationships within information, failing to discover valuable knowledge and patterns from massive amounts of multi-source data. For instance, in the financial sector, complex potential relationships exist between multiple sources of information, such as stock market trading data, news reports, and macroeconomic indicators, but existing technologies struggle to effectively reveal these relationships, thus affecting the accuracy of investment decisions. Therefore, there is an urgent need for a multimodal data processing method capable of integrating multi-source information and uncovering potential relationships. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a multimodal data processing method and system based on knowledge graphs.

[0005] In conjunction with the first aspect of this application, a knowledge graph-based multimodal data processing method is provided, applied to a knowledge graph-based multimodal data processing system, the method comprising:

[0006] Obtain a multi-source information set, which includes information units in different forms of expression and the source identifier of each information unit;

[0007] The multi-source information set is mapped to a preset semantic association network to generate a set of information units with semantic association tags, which are used to indicate the conceptual association relationship between the information units.

[0008] Cross-source feature association processing is performed on the set of information units with semantic association labels to obtain a comprehensive feature set that integrates semantic associations;

[0009] The semantic reasoning rules are invoked to perform association expansion processing on the comprehensive feature set, generating an expanded feature set containing potential association relationships;

[0010] Based on the extended feature set, information processing results that conform to the preset application scenario are generated, and the information processing results are fed back to the corresponding application interface.

[0011] In conjunction with the second aspect of this application, a knowledge graph-based multimodal data processing system is provided. The knowledge graph-based multimodal data processing system includes a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the knowledge graph-based multimodal data processing system implements the aforementioned knowledge graph-based multimodal data processing method.

[0012] In conjunction with a third aspect of this application, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when the computer-executable instructions are executed, the aforementioned knowledge graph-based multimodal data processing method is implemented.

[0013] Combining any of the above aspects, by acquiring a multi-source information set containing information units with different expressions and source identifiers, and using a pre-defined semantic association network for entity mapping, semantic association labels can be accurately assigned to information units. This establishes conceptual relationships between information units at the semantic level. Cross-source feature association processing is then performed on the information unit set with semantic association labels, further uncovering potential feature connections between information from different sources. This yields a comprehensive feature set integrating semantic associations, enriching the feature dimensions of the information and improving its completeness and accuracy. Calling semantic reasoning rules to extend the association of the comprehensive feature set automatically discovers potential relationships within the information, generating an extended feature set containing these potential relationships, significantly expanding the value and application scope of the information. Finally, information processing results conforming to a pre-defined application scenario are generated based on the extended feature set, and the results are fed back to the corresponding application interface, significantly improving the efficiency and quality of information processing. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained in conjunction with these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating the knowledge graph-based multimodal data processing method provided in this application embodiment. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0018] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0019] Figure 1 This illustration shows a flowchart of a knowledge graph-based multimodal data processing method provided in an embodiment of this application. It should be understood that in other embodiments, the order of some steps in this knowledge graph-based multimodal data processing method may be shared based on actual needs, or some steps may be omitted or maintained. The detailed components of this knowledge graph-based multimodal data processing method are as follows:

[0020] Step S110: Obtain a multi-source information set, which includes information units in different forms of expression and the source identifier of each information unit.

[0021] Acquiring multi-source information sets requires encompassing various forms of expression, including but not limited to text, images, audio, and video. Each information unit is a fundamental component of a multi-source information set, possessing independent content and a clearly defined scope.

[0022] Source identifiers are key information for distinguishing the sources of different information units, and their composition should include sufficient elements for traceability. For information units in text form, the source identifier can be composed of the system name that generated the text, the timestamp of the text generation, and the unique number of the text within that system; for information units in image form, the source identifier can include information such as the device model that took the image, the time of taking the image, the unique identifier of the device in the network, and the number of the image stored in the device's memory.

[0023] During the acquisition of information units, a preliminary integrity check is required for each unit. For text-based information units, check for issues such as garbled characters or missing key paragraphs; for image-based information units, check for image corruption or resolution not meeting requirements; for audio-based information units, check for excessive background noise or abnormal audio duration. Once a problematic information unit is found, its source identifier should be recorded promptly, and the relevant information should be reported to the information provider for re-acquisition or correction.

[0024] For information units involving privacy-sensitive data, appropriate privacy protection technologies must be adopted. For example, for text information units containing sensitive personal identification information, de-identification processing is used to replace the sensitive information with specific identifiers; for image information units containing parts involving personal privacy, blurring or feature extraction is used to remove privacy areas from the original image; during data transmission, encrypted transmission protocols are used to ensure that data is not leaked during transmission; during data storage, encrypted storage methods are used to encrypt the stored data, and only authorized access can decrypt and obtain the original data.

[0025] Step S120: Perform entity mapping processing on the multi-source information set and the preset semantic association network to generate a set of information units with semantic association labels, wherein the semantic association labels are used to indicate the conceptual association relationship between the information units.

[0026] The pre-defined semantic association network is a structured knowledge system containing a large number of entities and the relationships between them. Entities can be various concrete things, abstract concepts, events, etc., while the relationships between entities describe the semantic connections between them. The purpose of entity mapping is to establish corresponding relationships between information units in a multi-source information set and entities in the semantic association network, thereby assigning each information unit a semantic association label that indicates its conceptual relationship with other information units.

[0027] Step S121: Analyze each information unit in the multi-source information set and extract the core concepts and concept attribute descriptions in the information unit.

[0028] For text-based information units, the parsing process requires the use of natural language processing techniques. First, word segmentation is performed to divide the text information units into individual words or phrases. Next, part-of-speech tagging is performed to determine the part of speech of each word or phrase. Then, named entity recognition is performed to identify entities with specific meanings in the text. Finally, core concepts are extracted from the identified entities, and the attributes of these core concepts are analyzed to form a concept attribute description.

[0029] For information units in the form of images, the parsing process requires the use of computer vision technology. First, the image is preprocessed, including image denoising, image enhancement, and image resizing. Then, feature extraction is performed, extracting color, texture, and shape features from the image. Next, image recognition algorithms are used to identify objects and scenes in the image. Finally, the core concepts in the image are determined, and their conceptual attribute descriptions are extracted.

[0030] The parsing of audio information units requires the use of audio processing techniques. First, the audio undergoes preprocessing, including noise reduction, filtering, and audio segmentation. Then, feature extraction is performed, extracting spectral features and Mel-frequency cepstral coefficients. Next, speech recognition technology converts the audio into text, which is then processed. Finally, the core concepts and their attribute descriptions are determined.

[0031] The parsing process of video information units requires the combination of computer vision and audio processing technologies. Each frame of the video image is processed to extract the core concepts and their attributes; simultaneously, the audio portion of the video is processed to extract the core concepts and their attributes; finally, the processing results of the images and audio are combined to determine the core concepts and their attribute descriptions for the video information unit.

[0032] Step S122: Perform similarity comparison processing between the core concept and entity nodes in the preset semantic association network to determine the target entity node and matching degree parameter that matches the core concept.

[0033] First, the core concepts and entity nodes in the semantic association network need to be converted into a form that can be used for similarity calculation. Word vector technology can be used to convert both core concepts and entity nodes into vector representations; the vector dimensions can be set according to actual needs.

[0034] Next, the similarity between the core concept vector and the entity node vector is calculated. Commonly used similarity calculation methods include cosine similarity and Euclidean distance. Cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them; the closer the cosine value is to 1, the higher the similarity between the two vectors. Euclidean distance measures the similarity by calculating the straight-line distance between two vectors in space; the smaller the distance, the higher the similarity between the two vectors.

[0035] Based on the calculated similarity, the matching degree parameter is determined. The range of values ​​for the matching degree parameter can be set according to the actual situation. Generally speaking, the larger the value, the higher the matching degree between the core concept and the entity node.

[0036] Step S123: Filter out target entity nodes whose matching degree meets the preset conditions according to the matching degree parameter, and obtain the set of associated nodes and the association type description of the target entity nodes in the semantic association network.

[0037] The preset condition can be set to a matching degree parameter greater than or equal to a preset threshold. This threshold can be set according to the actual application scenario and the characteristics of the semantic association network. Entity nodes whose matching degree parameter meets the preset condition will be selected as target entity nodes.

[0038] For each selected target entity node, all associated entity nodes are searched in the semantic association network to form a set of associated nodes. Simultaneously, a description of the association type between each associated node and the target entity node is recorded; this description must clearly indicate the semantic relationship between the two.

[0039] Step S124: Generate semantic association tags for corresponding information units based on the set of associated nodes and the association type description. The semantic association tags include the main entity identifier, the associated entity identifier, and the association type identifier.

[0040] The primary entity identifier is a unique identifier for the target entity node within the semantic association network, used to distinguish different entity nodes. The associated entity identifier is a unique identifier for each associated node in the set of associated nodes. The association type identifier is an encoded representation of the association type description, designed to facilitate computer processing and recognition of association types.

[0041] When generating semantic association tags, the main entity identifier, associated entity identifier, and association type identifier need to be organized according to a certain format. For example, the format "Main entity identifier|Associated entity identifier 1-Associated type identifier 1|Associated entity identifier 2-Associated type identifier 2|..." can be used.

[0042] Step S125: Bind each information unit to its corresponding semantic association tag to form a set of information units with semantic association tags.

[0043] Binding can be achieved by establishing a mapping relationship between information units and semantic association tags. Each information unit can be assigned a unique identifier, and the semantic association tag can also be associated with this identifier. In this way, the corresponding information unit and semantic association tag can be quickly found through this identifier.

[0044] By integrating all the bound information units together, a set of information units with semantic association tags is formed.

[0045] Step S130: Perform cross-source feature association processing on the set of information units with semantic association labels to obtain a comprehensive feature set with fused semantic associations.

[0046] The purpose of cross-source feature association processing is to associate and fuse features of information units from different sources, making full use of the complementarity between information from different sources, thereby improving the expressive power of features.

[0047] Step S131: Group the set of information units with semantic association tags according to the source identifier of the information units to obtain multiple source feature groups. Each source feature group contains information units from the same source and their corresponding semantic association tags.

[0048] Iterate through the set of information units with semantic association tags and extract the source identifier for each information unit. Based on the different source identifiers, assign the information units to different groups. Information units with the same source identifier are grouped into the same source feature group.

[0049] Each source feature group contains not only the content of the information unit, but also the semantic association label corresponding to that information unit. This allows for feature association by combining semantic information in subsequent processing.

[0050] Step S132: Extract the intrinsic features of the information unit in each source feature group and the associated features in the semantic association tags. The intrinsic features are the descriptive features carried by the information unit itself, and the associated features are the entity association features contained in the semantic association tags.

[0051] For each source feature group, extract the intrinsic features for each information unit within it. The method for extracting intrinsic features depends on the type of information unit. For example, the intrinsic features of textual information units may be the frequency of keyword occurrences, sentence length, sentiment, etc.; the intrinsic features of image-based information units may be color histograms, texture feature vectors, shape parameters, etc.; and the intrinsic features of audio-based information units may be spectral features, audio duration, volume, etc.

[0052] Simultaneously, association features are extracted from semantic association tags. Association features include information such as main entity identifier, associated entity identifier, and association type identifier, which reflect the association relationships between the entities involved in the information unit and other entities.

[0053] Step S133: Perform feature space alignment processing on the intrinsic features of feature groups from different sources so that the intrinsic features from different sources are in the same feature dimension space.

[0054] Step S1331: Extract the dimensional information and feature distribution parameters of the intrinsic features of each source feature group, wherein the feature distribution parameters include the feature mean and feature variance.

[0055] For each source feature group, count the number of dimensions of its intrinsic features, which is the length of the feature vector. At the same time, calculate the feature mean and feature variance for each dimension, where the feature mean is the average of all feature values ​​in that dimension, and the feature variance is the average of the squared differences between all feature values ​​in that dimension and the mean.

[0056] Step S1332: Based on the dimensional information of the intrinsic features of all source feature groups, determine the maximum feature dimension value, and perform dimensional expansion processing on the intrinsic features that are lower than the maximum feature dimension value. The expanded feature dimension is consistent with the maximum feature dimension value.

[0057] Compare the intrinsic feature dimensions of all source feature groups and find the largest dimension value. For intrinsic features with dimensions lower than this largest dimension value, dimensional expansion processing is required.

[0058] Dimension expansion can be achieved by zero-padding, which involves adding zeros to the end of the feature vector to maximize its dimensionality.

[0059] Step S1333: Calculate the deviation between the feature distribution parameters of the intrinsic features of each source feature group and the preset standard feature distribution parameters, wherein the preset standard feature distribution parameters are the pre-set feature mean and feature variance.

[0060] The preset standard feature distribution parameters are obtained statistically from a large amount of sample data, including the standard feature mean and standard feature variance. For each dimension of each source feature group, the difference between its feature mean and the standard feature mean, and the difference between its feature variance and the standard feature variance are calculated. These differences are the bias values.

[0061] Step S1334: Construct a feature mapping function based on the deviation value. The feature mapping function is used to map intrinsic features from different sources to the feature space corresponding to the preset standard feature distribution parameters.

[0062] The construction of feature mapping functions needs to consider the impact of bias values ​​on features, and the bias can be eliminated by adjusting the feature values. A linear mapping approach can be used. For each dimension, the feature mapping function can be expressed as the mapped feature value equal to the original feature value minus the mean bias, multiplied by the square root of the ratio of the standard feature variance to the feature variance of that dimension.

[0063] Step S1335: Input the intrinsic features of each source feature group into the feature mapping function for mapping processing to obtain aligned intrinsic features in the same feature dimension space.

[0064] The feature values ​​of each dimension of the intrinsic features after dimensional expansion are input into the corresponding feature mapping function to calculate the mapped feature values. Once all dimensions have been mapped, the resulting feature vector is the aligned intrinsic feature. At this point, the intrinsic features of feature groups from different sources are in the same feature dimension space and have similar feature distributions.

[0065] Step S134: Based on the association type identifier in the semantic association label, calculate the association strength between the aligned intrinsic features and the associated features to generate the association weight matrix between features.

[0066] Step S1341: Parse the association type identifier in the semantic association tag and determine the preset association strength benchmark value corresponding to each association type.

[0067] The preset association strength benchmark value is set in advance based on the importance and commonality of the association type. Different association type identifiers correspond to different benchmark values.

[0068] Step S1342: Extract the common attribute descriptions of the aligned intrinsic features and associated features, and calculate the overlap parameter of the common attribute descriptions. The overlap parameter is the ratio of the number of common attribute descriptions to the total number of attribute descriptions.

[0069] Both aligned intrinsic and associated features have their own attribute descriptions. We extract the common attribute descriptions and count their number. Simultaneously, we calculate the total number of attribute descriptions for both intrinsic and associated features, which is the sum of the number of attribute descriptions for intrinsic features and associated features, minus the number of common attribute descriptions. The overlap parameter is equal to the ratio of the number of common attribute descriptions to the total number of attribute descriptions.

[0070] Step S1343: Calculate the initial value of the association strength based on the preset association strength benchmark value and the overlap parameter. The initial value of the association strength is positively correlated with the preset association strength benchmark value and the overlap parameter.

[0071] The initial value of the association strength can be obtained by multiplying the preset association strength benchmark value by the overlap parameter.

[0072] Step S1344: Normalize the initial value of the association strength, arrange the normalized association strength value into a matrix according to the entity association identifier corresponding to the feature, and generate an association weight matrix between features. The rows and columns of the association weight matrix correspond to different feature elements, and the matrix element value is the association strength value between the corresponding two feature elements.

[0073] Normalization can map the initial values ​​of association strength to a range between 0 and 1. This can be achieved by dividing all initial values ​​by the maximum value.

[0074] Then, based on the entity association identifiers corresponding to the features, the row and column indices of each feature in the matrix are determined, and the normalized association strength values ​​are filled into the corresponding matrix positions to form an association weight matrix.

[0075] Step S135: Perform weighted fusion processing on the features of different source feature groups according to the association weight matrix to generate a comprehensive feature set with fused semantic association. Each feature element in the comprehensive feature set contains a source identifier, an entity association identifier, and a feature value description.

[0076] The weighted fusion process uses a weighted concatenation method, combining features from different source feature groups according to their weights in the association weight matrix. For each feature element, it is concatenated with other relevant features based on its weight in the association weight matrix.

[0077] By integrating all the merged feature elements together, a comprehensive feature set with fused semantic associations is formed.

[0078] Step S140: Invoke semantic reasoning rules to perform association expansion processing on the comprehensive feature set, and generate an expanded feature set containing potential association relationships.

[0079] By invoking semantic reasoning rules, potential relationships are extracted from the comprehensive feature set, thereby enriching the content of the feature set.

[0080] Step S141: Parse the entity association identifiers in the comprehensive feature set to determine the existing entity association relationships and association path information.

[0081] The system iterates through the comprehensive feature set, extracts entity association identifiers, analyzes the relationships between these identifiers, and determines existing entity associations. Simultaneously, it organizes association path information, that is, the order and connection method of associations between entities.

[0082] Step S142: Call the preset semantic reasoning rule library to extract reasoning rules that match the existing entity association relationship. The reasoning rules include premise association conditions and conclusion association relationships.

[0083] The pre-defined semantic reasoning rule base stores various reasoning rules, each containing premise association conditions and conclusion association relationships. Based on existing entity association relationships, the rule base is searched for matching reasoning rules, i.e., rules whose premise association conditions match existing association relationships.

[0084] Step S143: Compare the existing entity association relationships and association path information with the premise association conditions of the reasoning rules, and select the target reasoning rules that meet the premise association conditions.

[0085] Each existing entity relationship and association path information is compared with the premise association conditions of the inference rule to check whether they are fully met. The inference rule that meets the conditions is selected as the target inference rule.

[0086] Step S144: Apply the target reasoning rule to perform reasoning expansion processing on the existing entity association relationship to generate potential association relationship. The potential association relationship is an entity association relationship that is not directly reflected in the comprehensive feature set but is obtained through reasoning.

[0087] Based on the conclusions of the target reasoning rules, and combined with the existing entity relationships, we can deduce the relationships that are not directly reflected in the comprehensive feature set, i.e., potential relationships.

[0088] Step S1441: Extract entity attribute constraints and association path length constraints from the premise association conditions of the target inference rule.

[0089] From the premise association conditions of the target reasoning rule, constraints on entity attributes and association path length are extracted. Entity attribute constraints restrict the attributes possessed by the entities participating in the association, while association path length constraints restrict the length of the association path between entities.

[0090] Step S1442: Filter out the association fragments that meet the entity attribute constraints and association path length constraints from the existing entity associations.

[0091] Based on the extracted entity attribute constraints and association path length constraints, the existing entity associations are filtered to find association fragments that meet these two constraints.

[0092] Step S1443: Reorganize the association fragments according to the structure of the conclusion association of the target reasoning rule to generate preliminary potential associations.

[0093] Based on the structure of the conclusion associations according to the goal reasoning rules, the selected association fragments are reorganized to form preliminary potential associations.

[0094] Step S1444: Perform consistency verification on the preliminary potential relationship to check whether there is any contradiction between the preliminary potential relationship and the existing entity relationship.

[0095] Perform a consistency check on the initial potential relationships to see if there are any contradictions between them and the existing entity relationships.

[0096] Step S1445: If there is no contradiction, calculate the association credibility parameter of the preliminary potential association relationship. The association credibility parameter is calculated based on the credibility of the target inference rule and the matching degree of the association relationship fragment.

[0097] If the initial potential association does not contradict existing entity associations, then the association credibility parameter of the initial potential association is calculated. The calculation of the association credibility parameter is based on the credibility of the target inference rule and the matching degree of the association fragment.

[0098] Step S1446: Determine the preliminary potential associations whose association confidence parameters meet the preset threshold as the final potential associations.

[0099] Set a preset threshold, and determine the preliminary potential associations that reach the association credibility parameter as the final potential associations.

[0100] Step S145: Add the feature information corresponding to the potential relationship to the comprehensive feature set to form an extended feature set containing potential relationships. Each potential relationship in the extended feature set contains a reasoning basis identifier and a relationship credibility parameter.

[0101] The reasoning basis identifier can be a unique number of the target reasoning rule in the semantic reasoning rule base. This number allows for quick location of the corresponding reasoning rule, facilitating subsequent verification and tracing of potential associations. The value range of the association credibility parameter is consistent with the previous matching degree parameter, and its magnitude comprehensively reflects the reliability of the potential association.

[0102] By adding these potential relationship features, which include inference basis identifiers and association credibility parameters, to the comprehensive feature set, the content of the comprehensive feature set is expanded, forming an extended feature set. The extended feature set not only includes the features obtained after processing the original information units, but also incorporates the potential relationship features obtained through inference, making the feature set richer in information and able to more comprehensively reflect the various relationships between information units.

[0103] Step S150: Generate information processing results that conform to the preset application scenario based on the extended feature set, and feed the information processing results back to the corresponding application interface.

[0104] Step S151: Parse the requirement description of the preset application scenario and extract the core feature types and feature output format requirements required by the application scenario.

[0105] A thorough analysis of the requirements description for the pre-defined application scenario is conducted to clarify the types of core features needed in that scenario. Core feature types refer to feature categories closely related to the application scenario objectives. For example, some scenarios may require features indicating direct relationships between entities, while others may require features representing specific attributes.

[0106] Simultaneously, the output format requirements for extracted features include the structure, data type, and field names of the output data. Different application scenarios may have different requirements for data format; some may require a list format, some may require organization in a specific key-value pair format, and some may require a data exchange format that conforms to a certain standard.

[0107] Step S152: Select feature elements that match the core feature type from the extended feature set to form a scene adaptation feature set.

[0108] Step S1521: Convert the core feature types of the preset application scenario into feature filtering conditions, wherein the feature filtering conditions include feature attribute descriptions and feature association type descriptions.

[0109] Based on the extracted core feature types, specific feature selection criteria are formulated. The feature attribute description clarifies the attributes that the required features should possess, such as the value range of the features and the type of entity the features are associated with; the feature association type description specifies the type of association that should exist between features, such as whether it is a direct or indirect association, a causal association or a parallel association, etc.

[0110] Step S1522: Traverse each feature element in the extended feature set and check whether the attribute description and association type description of the feature element meet the feature filtering conditions.

[0111] Examine each feature element in the extended feature set one by one, compare its attribute description with the feature attribute description in the feature filtering conditions to see if they are consistent, and check if its association type description matches the feature association type description in the feature filtering conditions.

[0112] Step S1523: Mark the feature elements that meet the feature selection criteria as candidate feature elements, and record the association confidence parameters of the feature elements.

[0113] Feature elements whose attribute descriptions and association type descriptions both meet the feature selection criteria are marked as candidate feature elements. Simultaneously, the association confidence parameter of these candidate feature elements is recorded; this parameter will be used in subsequent sorting operations.

[0114] Step S1524: Sort the candidate feature elements from high to low according to the association confidence parameter to form a candidate feature sequence.

[0115] Candidate feature elements are sorted according to their association confidence parameter, with those having higher confidence parameters ranked first and those having lower confidence parameters ranked last, thus forming a candidate feature sequence. The purpose of this sorting is to prioritize feature elements with higher confidence, thereby improving the reliability of the subsequent information processing results.

[0116] Step S1525: Based on the feature quantity requirements of the preset application scenario, select the first few candidate feature elements from the candidate feature sequence to form a scenario-adaptive feature set. If the number of candidate feature elements is insufficient, issue a feature supplement prompt message.

[0117] Based on the feature quantity requirements of the preset application scenario, a certain number of top-ranked candidate feature elements are selected from the candidate feature sequence to form a scenario-adaptive feature set. If the total number of candidate feature elements is less than the preset feature quantity requirement, it indicates that the current expanded feature set cannot meet the needs of the application scenario. In this case, a feature supplementation prompt message needs to be issued to remind relevant personnel to supplement or process the features.

[0118] Step S153: Perform format conversion processing on the scene adaptation feature set according to the feature output format requirements, so that the converted feature set conforms to the data format standard of the application scenario.

[0119] Based on the feature output format requirements extracted in step S151, each feature element in the scenario-adaptive feature set undergoes format conversion. For example, the attribute values ​​of the feature elements are converted to the specified data type, the organization of the feature elements is adjusted to the required structure, and necessary field names are added to ensure that the converted feature set fully conforms to the data format standards of the application scenario.

[0120] Step S154: Perform redundant information removal processing on the feature set after format conversion, and delete duplicate or irrelevant feature elements.

[0121] Step S1541: Calculate the hash value of the feature elements in the feature set after format conversion. Each feature element corresponds to a unique hash value.

[0122] A hash algorithm is used to calculate a unique hash value for each feature element after format conversion. Since identical feature elements will generate the same hash value, duplicate feature elements can be identified by comparing the hash values.

[0123] Step S1542: Identify duplicate feature elements by comparing hash values, retain the feature element with the highest association confidence parameter, and delete other duplicate feature elements.

[0124] Compare the hash values ​​of all feature elements to identify those with the same hash value; these are the duplicate feature elements. Among these duplicate feature elements, retain the one with the highest association confidence parameter, and delete the rest to avoid information redundancy.

[0125] Step S1543: Parse the irrelevant feature identifiers in the application scenario requirement description, whereby the irrelevant feature identifiers are used to indicate feature types that are irrelevant to the application scenario.

[0126] Irrelevant feature identifiers are extracted from the application scenario requirements description. These identifiers clarify which types of features are irrelevant to the current application scenario requirements and do not need to be included in the final information processing results.

[0127] Step S1544: Compare the feature elements in the feature set with irrelevant feature identifiers, filter out feature elements belonging to irrelevant feature types, and delete them.

[0128] Each feature element in the feature set is compared with an irrelevant feature identifier to determine whether the feature element belongs to an irrelevant feature type. If it does, it is removed from the feature set to simplify its contents.

[0129] Step S1545: Check the correlation between the remaining feature elements. If there is a cyclical correlation and the feature element combination is meaningless to the application scenario, delete the feature element with the lowest correlation strength.

[0130] Analyze the relationships between the remaining feature elements to check for circular associations, i.e., feature element A is associated with feature element B, feature element B is associated with feature element C, and feature element C is associated with feature element A, forming a cycle. If the above circular associations are not meaningful for the application scenario, delete the feature element with the weakest association strength from the feature elements that constitute the circular association to optimize the structure of the feature set.

[0131] Step S1546: Perform an integrity check on the processed feature set to determine whether the retained feature elements meet the core requirements of the application scenario. If not, return to the feature filtering step to re-filter the feature elements.

[0132] The process assesses whether the processed feature set contains all the core features required by the application scenario and whether it meets the core requirements of the application scenario. If not, it returns to step S152 and repeats the feature filtering, sorting, and selection operations until the feature set meets the core requirements of the application scenario.

[0133] Step S155: Encapsulate the processed feature set into an information processing result, which includes feature data and data description information. The data description information is used to explain the source and correlation of the feature data.

[0134] The feature set, after redundancy removal and integrity verification, is encapsulated to form the information processing result. Feature data is the specific content of the processed feature set, while data description information explains in detail the source of each feature data, such as which original information unit it comes from, what processing steps it has undergone, and the relationships between feature data, helping users in the application scenario understand the meaning and background of the feature data.

[0135] Step S156: Determine the corresponding application interface based on the identification information of the preset application scenario, and send the information processing result to the application interface.

[0136] The preset application scenario identifier is a unique identifier used to distinguish different application scenarios. Based on this identifier, the corresponding application interface can be found. The application interface is the channel for outputting information processing results; different application scenarios may correspond to different application interfaces. The encapsulated information processing results are sent out through the designated application interface to complete the feedback of the information processing results.

[0137] In the above embodiments, the knowledge graph-based multimodal data processing system for performing the above method embodiments has at least one processor, a control module (chipset) coupled to at least one of the processors, a memory coupled to the control module, a non-volatile memory (NVM) / storage device coupled to the control module, at least one load to / output device coupled to the control module, and a network interface coupled to the control module.

[0138] The processor may include at least one single-core or multi-core processor, and may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). For some alternative implementations, a knowledge graph-based multimodal data processing system can serve as an electronic device such as a gateway as described in the embodiments of this application.

[0139] In some alternative implementations, a knowledge graph-based multimodal data processing system may include at least one computer-readable medium (e.g., a memory or NVM / storage device) having instructions and at least one processor fused with the at least one computer-readable medium and configured to execute the instructions to implement the module thereby performing the actions described in this disclosure.

[0140] In one embodiment, the control module may include any suitable interface controller to provide any suitable interface to at least one of the processors and / or any suitable device or component communicating with the control module.

[0141] The control module may include a memory controller module to provide an interface to the memory. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0142] The memory can be used, for example, to load and store data and / or instructions for a knowledge graph-based multimodal data processing system. In one embodiment, the memory may include any suitable volatile memory, such as suitable DRAM.

[0143] In one embodiment, the control module may include at least one load-to-output controller to provide an interface to the NVM / storage device and (at least one) load-to-output device.

[0144] For example, an NVM / storage device can be used to store data and / or instructions. An NVM / storage device may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (at least one) non-volatile storage device (e.g., at least one hard disk drive (HDD), at least one optical disc (CD) drive, and / or at least one digital universal optical disc (DVD) drive).

[0145] NVM / storage devices may include storage resources that are physically part of a device on which a knowledge graph-based multimodal data processing system is mounted, or that can be accessed by the device without being part of it. For example, an NVM / storage device may be accessed over a network via at least one load-to-output device.

[0146] At least one loading / output device may provide an interface for the knowledge graph-based multimodal data processing system to communicate with any other suitable device. The loading / output device may include communication components, phonetic components, sensor components, etc. A network interface may provide an interface for the knowledge graph-based multimodal data processing system to communicate over at least one network. The knowledge graph-based multimodal data processing system may wirelessly communicate with at least one component of a wireless network based on at least one wireless network prior and / or any prior and / or protocol, such as accessing a wireless network based on communication priors.

[0147] In one embodiment, at least one of the processors may be integrated with the logic of at least one controller of the control module (e.g., a memory controller module). In one embodiment, at least one of the processors may be integrated with the logic of at least one controller of the control module to form a system-level integration. In one embodiment, at least one of the processors may be fused with the logic of at least one controller of the control module on the same die. In one embodiment, at least one of the processors may be fused with the logic of at least one controller of the control module on the same die to form a system-on-a-chip (SoC).

[0148] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0149] This invention discloses a computer read storage medium that stores a computer program for electronic data interchange, wherein the computer program causes a computer to execute the steps in the knowledge graph-based multimodal data processing method described in the foregoing embodiments.

[0150] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the knowledge graph-based multimodal data processing method described in the foregoing embodiments.

[0151] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0152] Finally, it should be noted that the above-disclosed embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal data processing method based on knowledge graphs, characterized in that, The method includes: Obtain a multi-source information set, which includes information units in different forms of expression and the source identifier of each information unit; The multi-source information set is mapped to a preset semantic association network to generate a set of information units with semantic association tags, which are used to indicate the conceptual association relationship between the information units. Cross-source feature association processing is performed on the set of information units with semantic association labels to obtain a comprehensive feature set that integrates semantic associations; The semantic reasoning rules are invoked to perform association expansion processing on the comprehensive feature set, generating an expanded feature set containing potential association relationships; Based on the extended feature set, information processing results that conform to the preset application scenario are generated, and the information processing results are fed back to the corresponding application interface.

2. The multimodal data processing method based on knowledge graphs according to claim 1, characterized in that, The step of performing entity mapping processing between the multi-source information set and a preset semantic association network to generate a set of information units with semantic association tags includes: Analyze each information unit in the multi-source information set and extract the core concepts and concept attribute descriptions from the information unit; The core concept is compared with the entity nodes in the preset semantic association network to determine the target entity node and matching degree parameter that matches the core concept. Based on the matching degree parameter, target entity nodes that meet the preset conditions are selected, and the set of associated nodes and the description of the association type of the target entity node in the semantic association network are obtained. Based on the set of associated nodes and the description of the association type, a semantic association tag is generated for the corresponding information unit. The semantic association tag includes the main entity identifier, the associated entity identifier, and the association type identifier. Each information unit is bound to its corresponding semantic association tag to form a set of information units with semantic association tags.

3. The multimodal data processing method based on knowledge graphs according to claim 1, characterized in that, The cross-source feature association processing of the information unit set with semantic association labels yields a comprehensive feature set that integrates semantic associations, including: The set of information units with semantic association tags is grouped according to the source identifier of the information units to obtain multiple source feature groups. Each source feature group contains information units from the same source and their corresponding semantic association tags. Extract the intrinsic features of information units in each source feature group and the associated features in semantic association tags. The intrinsic features are the descriptive features carried by the information unit itself, and the associated features are the entity association features contained in the semantic association tags. The intrinsic features of feature groups from different sources are aligned in the feature space so that the intrinsic features from different sources are in the same feature dimension space. Based on the association type identifier in the semantic association label, the association strength is calculated between the aligned intrinsic features and associated features to generate an association weight matrix between features; The features from different source feature groups are weighted and fused according to the association weight matrix to generate a comprehensive feature set with fused semantic association. Each feature element in the comprehensive feature set contains a source identifier, an entity association identifier, and a feature value description.

4. The knowledge graph-based multimodal data processing method according to claim 3, characterized in that, The step of performing feature space alignment processing on the intrinsic features of feature groups from different sources, so that the intrinsic features from different sources are in the same feature dimension space, includes: Extract the dimensionality information and feature distribution parameters of the intrinsic features of each source feature group, wherein the feature distribution parameters include the feature mean and feature variance; Based on the dimensional information of the intrinsic features of all source feature groups, the maximum feature dimension value is determined, and the intrinsic features with dimensions lower than the maximum feature dimension value are expanded to match the maximum feature dimension value. Calculate the deviation between the feature distribution parameters of the intrinsic features of each source feature group and the preset standard feature distribution parameters, wherein the preset standard feature distribution parameters are the pre-set feature mean and feature variance; A feature mapping function is constructed based on the deviation value. The feature mapping function is used to map intrinsic features from different sources to the feature space corresponding to preset standard feature distribution parameters. The intrinsic features of each source feature group are input into the feature mapping function for mapping processing to obtain aligned intrinsic features in the same feature dimension space.

5. The multimodal data processing method based on knowledge graphs according to claim 3, characterized in that, The method of calculating the association strength between aligned intrinsic features and associated features based on the association type identifier in the semantic association label, and generating an association weight matrix between features, includes: Parse the association type identifier in the semantic association tags and determine the preset association strength benchmark value corresponding to each association type; Extract the common attribute descriptions of the aligned intrinsic features and associated features, and calculate the overlap parameter of the common attribute descriptions. The overlap parameter is the ratio of the number of common attribute descriptions to the total number of attribute descriptions. The initial value of the association strength is calculated based on the preset association strength benchmark value and the overlap parameter. The initial value of the association strength is positively correlated with both the preset association strength benchmark value and the overlap parameter. The initial value of the association strength is normalized, and the normalized association strength value is arranged in a matrix according to the entity association identifier corresponding to the feature to generate the association weight matrix between features. The rows and columns of the association weight matrix correspond to different feature elements, and the matrix element value is the association strength value between the corresponding two feature elements.

6. The multimodal data processing method based on knowledge graphs according to claim 1, characterized in that, The semantic reasoning rules are invoked to perform association expansion processing on the comprehensive feature set, generating an expanded feature set containing potential associations, including: Analyze the entity association identifiers in the comprehensive feature set to determine existing entity association relationships and association path information; Call the preset semantic reasoning rule library to extract reasoning rules that match the existing entity association relationship. The reasoning rules include premise association conditions and conclusion association relationships. The existing entity associations and association path information are compared with the premise association conditions of the reasoning rules to select target reasoning rules that meet the premise association conditions. The target reasoning rule is applied to the existing entity relationship to perform reasoning expansion processing to generate potential relationship. The potential relationship is the entity relationship that is not directly reflected in the comprehensive feature set but is obtained through reasoning. The feature information corresponding to the potential relationships is added to the comprehensive feature set to form an extended feature set containing potential relationships. Each potential relationship in the extended feature set contains a reasoning basis identifier and a relationship credibility parameter.

7. The multimodal data processing method based on knowledge graphs according to claim 6, characterized in that, The application of the target reasoning rule to extend the reasoning of existing entity relationships and generate potential relationships includes: Extract entity attribute constraints and association path length constraints from the preconditions of the target reasoning rule; Filter out relationship fragments from existing entity relationships that meet the entity attribute constraints and relationship path length constraints; The aforementioned relationship fragments are reorganized according to the structure of the conclusion relationship of the target reasoning rule to generate preliminary potential relationships; The preliminary potential relationships are subjected to consistency verification to check whether there are any contradictions between the preliminary potential relationships and the existing entity relationships; If there is no contradiction, the association credibility parameter of the preliminary potential association is calculated. The association credibility parameter is calculated based on the credibility of the target inference rule and the matching degree of the association fragment. Preliminary potential associations whose association credibility parameters meet the preset thresholds are identified as final potential associations.

8. The multimodal data processing method based on knowledge graphs according to claim 1, characterized in that, The step of generating information processing results that conform to a preset application scenario based on the extended feature set, and feeding back the information processing results to the corresponding application interface, includes: Analyze the requirements description of the preset application scenario, and extract the core feature types and feature output format requirements required by the application scenario; Feature elements that match the core feature type are selected from the extended feature set to form a scene-adaptive feature set; The scene-adaptive feature set is converted according to the feature output format requirements so that the converted feature set conforms to the data format standard of the application scenario. Redundant information is removed from the feature set after format conversion, and duplicate or irrelevant feature elements are deleted. The processed feature set is encapsulated into an information processing result, which includes feature data and data description information. The data description information is used to explain the source and correlation of the feature data. The corresponding application interface is determined based on the identification information of the preset application scenario, and the information processing result is sent to the application interface.

9. The multimodal data processing method based on knowledge graphs according to claim 8, characterized in that, The step of selecting feature elements from the extended feature set that match the core feature type to form a scene adaptation feature set includes: The core feature types of the preset application scenario are converted into feature filtering conditions, which include feature attribute descriptions and feature association type descriptions. Traverse each feature element in the extended feature set and check whether the attribute description and association type description of the feature element meet the feature filtering conditions; Feature elements that meet the feature selection criteria are marked as candidate feature elements, and the association confidence parameters of the feature elements are recorded. Candidate feature elements are sorted from high to low according to the association confidence parameter to form a candidate feature sequence; Based on the feature quantity requirements of the preset application scenario, select the first few candidate feature elements from the candidate feature sequence to form a scenario-adaptive feature set. If the number of candidate feature elements is insufficient, issue a feature supplement prompt message. Furthermore, the process of removing redundant information from the feature set after format conversion, and deleting duplicate or irrelevant feature elements, includes: Calculate the hash value of each feature element in the feature set after format conversion, and each feature element corresponds to a unique hash value. Duplicate feature elements are identified by comparing hash values. The feature element with the highest correlation confidence parameter is retained, and other duplicate feature elements are deleted. Parse irrelevant feature identifiers in the application scenario requirement description, where the irrelevant feature identifiers are used to indicate feature types that are irrelevant to the application scenario; The feature elements in the feature set are compared with irrelevant feature identifiers, and feature elements belonging to irrelevant feature types are filtered out and deleted. Check the relationships between the remaining feature elements. If there are circular relationships and feature element combinations that are meaningless to the application scenario, delete the feature element with the lowest relationship strength. The integrity of the processed feature set is checked to determine whether the retained feature elements meet the core requirements of the application scenario. If not, the process returns to the feature filtering step to re-filter the feature elements.

10. A multimodal data processing system based on knowledge graphs, characterized in that, The method includes a processor and a computer-readable storage medium storing machine-executable instructions, which, when executed by a computer, implement the knowledge graph-based multimodal data processing method according to any one of claims 1-9.

Images