Data association method and device, computer device and readable storage medium

By extracting multimodal data labels and using error correction models to correct the entity connection relationship of the knowledge graph, the problem of multimodal data integration and insufficient accuracy in the prior art is solved, and the construction accuracy of the knowledge graph is improved.

CN118982065AInactive Publication Date: 2024-11-19JIANGSU ZHONGQI YIRONG DATA TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411462795.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-19
Publication Date
2024-11-19
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing knowledge graph construction methods lack the integration and anomaly detection mechanism for multimodal data, resulting in low accuracy in building knowledge graphs, especially when processing text, image and video data.

Method used

By extracting the labels of multimodal data, an initial knowledge graph is constructed, and the connection relationship of entities is corrected using an error correction model to generate the corrected knowledge graph to improve the accuracy of building the knowledge graph.

Benefits of technology

The integration of multimodal data and the accuracy of the knowledge graph are achieved, and the accuracy of the knowledge graph is improved through the correction model to correct the entity connection relationship.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982065A_ABST
    Figure CN118982065A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of data processing, and discloses a data association method and device, a computer device and a readable storage medium, the data association method comprises the following steps: step S101, extracting labels of M pieces of multi-modal data; step S102, constructing a first knowledge graph according to the labels of the M pieces of multi-modal data; s103, inputting the first knowledge graph into the error correction model, and outputting a first adjacent matrix; s104, correcting the first knowledge graph according to the first adjacent matrix to obtain a second knowledge graph; according to the method, the integration of the multi-modal data is realized by extracting the labels of the multi-modal data, and the connection relationship of the entities in the knowledge graph is corrected by using the error correction model, so that the accuracy of constructing the knowledge graph is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a data association method, device, computer device and readable storage medium. Background Art

[0002] In the digital age, big data technology has led to the generation of massive amounts of data. Although the amount of data is huge, the value of isolated data points is limited. In order to extract more value from it, data association becomes the key, and knowledge graphs (KnowledgeGraph) have emerged as an efficient form of data association. Knowledge graphs structure data through the connection between entities. They can not only store data, but also express the semantic connection between entities, which helps to improve data mining capabilities.

[0003] The existing knowledge graph construction method usually first extracts entities from text data through named entity recognition technology, which can be HMM (hidden Markov model), LSTM (long short-term memory network model), etc., and then extracts the connection relationship between entities through a neural network model, which can be a Transformer model, etc.

[0004] However, existing knowledge graph construction methods lack the integration of multimodal data, which includes image data and video data in addition to text data. In addition, existing knowledge graph construction methods lack anomaly detection mechanisms. For example, the connection relationships between entities are abnormal or missing, resulting in low accuracy in constructing knowledge graphs. Summary of the invention

[0005] The present invention provides a data association method, device, computer device and readable storage medium to solve the technical problems in the above-mentioned background technology.

[0006] The present invention provides a data association method, comprising the following steps: Step S101, extracting labels of M multimodal data; Multimodal data includes: text data, image data and video data; Labels include: primary labels and secondary labels; The first-level labels are represented by words, and the second-level labels are represented by text summaries; Step S102, constructing a first knowledge graph according to the labels of the M multimodal data; The first knowledge graph includes: M entities, initial features of entities and connection relationships between entities; Each entity is assigned a unique code; The M entities are associated with the unique codes of the M multimodal data respectively; The initial features of an entity are represented by its label; The connection relationship between entities includes: if the similarity of the initial features corresponding to any two entities is greater than or equal to the similarity threshold, then there is a connection relationship between the two entities; Step S103, inputting the first knowledge graph into the error correction model, and outputting a first adjacency matrix; The element value of the jth column of the i-th row of the first adjacency matrix indicates whether there is a connection relationship between the i-th entity and the j-th entity of the first knowledge graph. If there is a connection relationship, the element value of the jth column of the i-th row of the first adjacency matrix is ​​assigned to 1, otherwise it is assigned to 0, where 1≤i≤M, 1≤j≤M; Step S104, correcting the first knowledge graph according to the first adjacency matrix to obtain a second knowledge graph; The representation of the second knowledge graph is the same as that of the first knowledge graph.

[0007] Furthermore, M, the first-level label, and the similarity threshold are all custom parameters.

[0008] Furthermore, extracting labels of multimodal data includes the following steps: Step S201: for text data, firstly, the text data is segmented by a word segmenter, then the segmented words are converted into word vector representations by a word vector model, and finally the word vector representations of multiple segmented words are input into a semantic analysis model to output a primary label and a secondary label; Step S202: for the image data, obtaining the primary label and the secondary label of the image data through the image-to-text model; Step S203: for the video data, firstly perform frame processing on the video data at equal time intervals to obtain A frame images, then convert the A frame images into grayscale histogram representation, and calculate the Bhattacharyya distance of adjacent grayscale histograms, and if it is determined that the Bhattacharyya distance is greater than or equal to a preset distance threshold, then use the grayscale image of the next frame as a key frame image, until an image sequence consisting of B key frame images is obtained, and finally input the image sequence into a time series analysis model to obtain the primary label and secondary label of the video data, wherein A and B are both custom parameters; The grayscale histogram is represented by a vector with 256 dimensions. The value of each dimension represents the number of pixels with the same grayscale level as the dimension index. The calculation formula of the Bhattacharyya distance between adjacent grayscale histograms is as follows: ; Where 1≤a≤A, Represents the Bhattacharyya distance between the a-th grayscale histogram and the a-1-th grayscale histogram, and Represent the values ​​of the b-th dimension of the a-th and a-1-th grayscale histograms respectively.

[0009] Furthermore, the calculation formula of the similarity Sim of the initial features corresponding to any two entities is as follows: ; Where 1≤i≤M, 1≤j≤M, i≠j, and Represent the word vector representation of the first-level label of the i-th entity and the j-th entity, and Represent the word vector representation of the secondary label of the i-th entity and the j-th entity respectively, represents the Euclidean distance between the word vector representations of the first-level labels of the i-th entity and the j-th entity, The Manhattan distance between the word vector representations of the secondary labels of the i-th entity and the j-th entity, and Both are custom parameters, and their sum is 1.

[0010] Furthermore, the error correction model includes: a conversion layer, a splicing layer, a padding layer, and a hidden layer; The conversion layer is used to convert the primary label and secondary label corresponding to the initial features of each entity into word vector representation through the word vector model; The concatenation layer is used to concatenate the word vector representations corresponding to the primary label and the secondary label of each entity to obtain a combined vector; The padding layer is used to unify the number of dimensions of the combined vector of each entity to the maximum number of dimensions of the combined vector of the entity. For word vectors whose number of dimensions is less than the maximum number of dimensions, the average value of the remaining dimensions is taken to fill them. The hidden layer inputs the first knowledge graph and outputs the first adjacency matrix.

[0011] Furthermore, the calculation formula of the hidden layer includes: ; ; ; Where 1≤u≤M, S represents the first adjacency matrix of the hidden layer output, represents the set of entities that have a connection relationship with the u-th entity, and Respectively represent the combination vectors of the u-th entity and the v-th entity, represents the update vector of the u-th entity, and Respectively represent the weight parameter and distance coefficient between the u-th entity and the v-th entity, represents the Manhattan distance between the u-th entity and the v-th entity, represents the weight parameter corresponding to the u-th entity, and denote the first weight parameter and the second weight parameter respectively, represents the bias parameter, It indicates that the update vectors of M entities are stacked, T indicates the transpose operation, exp indicates the power operation of a natural constant, and sigmoid indicates the sigmoid activation function.

[0012] Furthermore, according to step S101 to step S102, a first knowledge graph is obtained as training data of training samples for training the error correction model, and the connection relationship of entities in the first knowledge graph is corrected by manual labeling to obtain a second adjacency matrix, the second adjacency matrix and the first adjacency matrix have the same representation, the second adjacency matrix is ​​used as sample labels of training samples for training the error correction model, and the difference between the second adjacency matrix and the first adjacency matrix is ​​designated as the loss function for training the error correction model.

[0013] Furthermore, when the element value in the first adjacency matrix is ​​0, the connection relationship between the entities corresponding to the first knowledge graph is deleted, and when the element value in the first adjacency matrix is ​​1, the connection relationship between the entities corresponding to the first knowledge graph is established, until all element values ​​in the first adjacency matrix are traversed to obtain the second knowledge graph.

[0014] The present invention provides a data association device, comprising: A label extraction module, which is used to extract labels of M multimodal data; A first knowledge graph construction module, which is used to construct a first knowledge graph according to the labels of M multimodal data; A first knowledge graph error correction module, which is used to input the first knowledge graph into the error correction model and output a first adjacency matrix; The second knowledge graph generation module is used to correct the first knowledge graph according to the first adjacency matrix to obtain the second knowledge graph.

[0015] The present invention provides a readable storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the above-mentioned data association method is executed.

[0016] The beneficial effects of the present invention are as follows: the present invention realizes the integration of multimodal data by extracting labels of multimodal data, and uses an error correction model to correct the connection relationship of entities in the knowledge graph, thereby improving the accuracy of constructing the knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of a data association method of the present invention; Figure 2 is a flow chart of extracting labels of multimodal data of the present invention; Figure 3 It is a schematic diagram of a data association device of the present invention.

[0018] In the figure: label extraction module 301, first knowledge graph construction module 302, first knowledge graph error correction module 303, second knowledge graph generation module 304. DETAILED DESCRIPTION

[0019] The subject matter described herein will now be discussed with reference to example embodiments. It should be understood that the discussion of these embodiments is only to enable those skilled in the art to better understand and implement the subject matter described herein, and the functions and arrangements of the elements discussed may be changed without departing from the scope of protection of the contents of this specification. Each example may omit, replace or add various processes or components as needed. In addition, the features described relative to some examples may also be combined in other examples.

[0020] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in one or more embodiments of the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in one or more embodiments of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] like Figure 1 to Figure 3 As shown, a data association method comprises the following steps: Step S101, extracting labels of M multimodal data; Multimodal data includes: text data, image data and video data; Labels include: primary labels and secondary labels; The first-level labels are represented by words, and the second-level labels are represented by text summaries; Step S102, constructing a first knowledge graph according to the labels of the M multimodal data; The first knowledge graph includes: M entities, initial features of entities and connection relationships between entities; Each entity is assigned a unique code; The M entities are associated with the unique codes of the M multimodal data respectively; The initial features of an entity are represented by its label; The connection relationship between entities includes: if the similarity of the initial features corresponding to any two entities is greater than or equal to the similarity threshold, then there is a connection relationship between the two entities; Step S103, inputting the first knowledge graph into the error correction model, and outputting a first adjacency matrix; The element value of the jth column of the i-th row of the first adjacency matrix indicates whether there is a connection relationship between the i-th entity and the j-th entity of the first knowledge graph. If there is a connection relationship, the element value of the jth column of the i-th row of the first adjacency matrix is ​​assigned to 1, otherwise it is assigned to 0, where 1≤i≤M, 1≤j≤M; Step S104, correcting the first knowledge graph according to the first adjacency matrix to obtain a second knowledge graph; The representation of the second knowledge graph is the same as that of the first knowledge graph.

[0022] In one embodiment of the present invention, M and the similarity threshold are both custom parameters, M is set according to the amount of multimodal data, preferably, the similarity threshold is set to 0.8, and the first-level label is a custom parameter. For example, the first-level label can be an industry type or a data source type. The industry type can be agriculture, forestry, animal husbandry, and pharmaceutical industry, etc. The data source type can be news articles, social media, and academic papers. In addition, the first-level labels can be subdivided. For example, for academic papers, the first-level labels can be refined into artificial intelligence, deep learning, and machine vision, etc.

[0023] In one embodiment of the present invention, Figure 2 As shown, extracting labels for multimodal data includes the following steps: Step S201: for text data, firstly, the text data is segmented by a word segmenter, then the segmented words are converted into word vector representations by a word vector model, and finally the word vector representations of multiple segmented words are input into a semantic analysis model to output a primary label and a secondary label; Step S202: for the image data, obtaining the primary label and the secondary label of the image data through the image-to-text model; Step S203: for the video data, firstly perform frame processing on the video data at equal time intervals to obtain A frame images, then convert the A frame images into grayscale histogram representation, and calculate the Bhattacharyya distance of adjacent grayscale histograms, and if it is determined that the Bhattacharyya distance is greater than or equal to a preset distance threshold, then use the grayscale image of the next frame as a key frame image, until an image sequence consisting of B key frame images is obtained, and finally input the image sequence into a time series analysis model to obtain the primary label and secondary label of the video data, wherein A and B are both custom parameters; The grayscale histogram is represented by a vector with 256 dimensions, and the value of each dimension represents the number of pixels with the same grayscale level as the dimension index; For example, the value of the first dimension is 20, that is, the dimension index is 1, indicating that there are 20 pixels with a gray level of 1. The calculation formula for the Bhattacharyya distance of adjacent gray histograms is as follows: ; Where 1≤a≤A, Represents the Bhattacharyya distance between the a-th grayscale histogram and the a-1-th grayscale histogram, and Represent the values ​​of the b-th dimension of the a-th and a-1-th grayscale histograms respectively.

[0024] It should be noted that the word segmenter can be jieba or THULAC word segmenter, etc.; the word vector model can be fastText or BERT model, etc.; the semantic analysis model can be Transformer or BERT model, etc.; the image-to-text model can be Show and Tell or Show Attend and Tell model, etc.; the timing analysis model can be LSTM or GRU model, etc.; when the number of key frame images is greater than B, the preset distance threshold is increased; when the number of key frame images is less than B, the preset distance threshold is reduced until the number of key frame images obtained is equal to B.

[0025] In one embodiment of the present invention, the unique code of the entity is generated by a snowflake algorithm, and the unique code of the entity can also be generated by a UUID.

[0026] In one embodiment of the present invention, the calculation formula of the similarity Sim of the initial features corresponding to any two entities is as follows: ; where 1≤i≤M, 1≤j≤M, i≠j, and Represent the word vector representation of the first-level label of the i-th entity and the j-th entity, and Represent the word vector representation of the secondary label of the i-th entity and the j-th entity respectively, represents the Euclidean distance between the word vector representations of the first-level labels of the i-th entity and the j-th entity, The Manhattan distance between the word vector representations of the secondary labels of the i-th entity and the j-th entity, and Both are custom parameters, and the sum of the two is 1. Preferably, and Set to 0.8 and 0.2 respectively.

[0027] It should be noted that the number of dimensions represented by the word vectors of the first-level labels of the i-th entity and the j-th entity must be the same, and the number of dimensions represented by the word vectors of the second-level labels of the i-th entity and the j-th entity must also be the same. When the word vectors of the first-level labels or second-level labels of two entities are different, the word vector with fewer dimensions can be padded with zero values ​​to align it with the word vector with more dimensions, or it can be padded by taking the average of the values ​​of the remaining dimensions.

[0028] In one embodiment of the present invention, the error correction model includes: a conversion layer, a splicing layer, a padding layer, and a hidden layer; The conversion layer is used to convert the primary label and secondary label corresponding to the initial features of each entity into word vector representation through the word vector model; The concatenation layer is used to concatenate the word vector representations corresponding to the primary label and the secondary label of each entity to obtain a combined vector; the filling layer is used to unify the number of dimensions of the combined vector of each entity to the maximum number of dimensions of the combined vector of the entity, and fill the word vectors whose number of dimensions is less than the maximum number of dimensions by taking the average value of the remaining dimensions; The hidden layer inputs the first knowledge graph and outputs the first adjacency matrix.

[0029] It should be noted that the word vector model of the conversion layer can be a fastText or BERT model, etc., which will not be elaborated here. For word vectors whose number of dimensions is less than the maximum number of dimensions, the average value of the remaining dimensions is taken to fill them. For example, the maximum number of dimensions is 128. If the number of dimensions of the entity's combined vector is 100, the average value of the values ​​of these 100 dimensions is taken to fill the remaining 28 dimensions.

[0030] In one embodiment of the present invention, the calculation formula of the hidden layer includes: ; ; ; Where 1≤u≤M, S represents the first adjacency matrix of the hidden layer output, represents the set of entities that have a connection relationship with the u-th entity, and Respectively represent the combination vectors of the u-th entity and the v-th entity, represents the update vector of the u-th entity, and Respectively represent the weight parameter and distance coefficient between the u-th entity and the v-th entity, represents the Manhattan distance between the u-th entity and the v-th entity, represents the weight parameter corresponding to the u-th entity, and denote the first weight parameter and the second weight parameter respectively, represents the bias parameter, It indicates that the update vectors of M entities are stacked, T indicates the transpose operation, exp indicates the power operation of a natural constant, and sigmoid indicates the sigmoid activation function.

[0031] It should be noted that the weight parameters and bias parameters of the error correction model are both learnable parameters. For example, if the size of the combination vector of the entity is 1×128, then the size of the vector obtained by transposing it is 128×1. The weight parameters of the entity and the weight parameters of the entity can be designed as a P×128 matrix. The weight parameters of the entity are multiplied by the transpose of the combination vector of the entity to obtain a P×1 vector. Similarly, the distance coefficient between entities is a real number. The weight parameters of the entity are multiplied by the distance coefficient between the entities and then multiplied by the transpose of the combination vector of the entity to obtain a P×1 vector. Therefore, the size of the update vector of the entity is P×1, then the update vectors of M entities are stacked to obtain an M×P matrix. The first weight parameter can be designed as a P×Q matrix. The M×P matrix is ​​multiplied by the first weight parameter to obtain an M×Q matrix, and the M×Q matrix is ​​multiplied by the Q×M matrix to obtain an M×M matrix, that is, the size of the first adjacency matrix output by the hidden layer is M×M, where P and Q are both artificially set parameters. Preferably, P is set to 16 and Q is set to 8.

[0032] In one embodiment of the present invention, a first knowledge graph is obtained as training data of training samples for training an error correction model according to step S101 to step S102, and the connection relationship of entities in the first knowledge graph is corrected by manual labeling to obtain a second adjacency matrix, the second adjacency matrix has the same representation as the first adjacency matrix, the second adjacency matrix serves as sample labels of training samples for training the error correction model, and the difference between the second adjacency matrix and the first adjacency matrix is ​​designated as the loss function for training the error correction model.

[0033] It should be noted that the gradient information of the loss function with respect to each weight parameter and bias parameter is calculated by the chain rule, and the weight parameters and bias parameters of the error correction model are updated according to the gradient information, so that the loss value obtained by the loss function calculation of the error correction model in each iteration is minimized. The training of the neural network model is a conventional technical means and will not be elaborated here.

[0034] In one embodiment of the present invention, when the element value in the first adjacency matrix is ​​0, the connection relationship between the entities corresponding to the first knowledge graph is deleted, and when the element value in the first adjacency matrix is ​​1, the connection relationship between the entities corresponding to the first knowledge graph is established until all element values ​​in the first adjacency matrix are traversed to obtain the second knowledge graph.

[0035] In one embodiment of the present invention, Figure 3 As shown, the present invention provides a data association device, comprising: A label extraction module 301 is used to extract labels of M multimodal data; A first knowledge graph construction module 302, which is used to construct a first knowledge graph according to the labels of M multimodal data; A first knowledge graph error correction module 303, which is used to input the first knowledge graph into the error correction model and output a first adjacency matrix; The second knowledge graph generation module 304 is used to correct the first knowledge graph according to the first adjacency matrix to obtain the second knowledge graph.

[0036] In one embodiment of the present invention, the present invention provides a computer device, which includes a display screen, a memory, a processor and a computer program, wherein the memory stores the computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned data association method.

[0037] In one embodiment of the present invention, the present invention provides a readable storage medium storing non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, the steps of the above-mentioned data association method are executed.

[0038] Furthermore, the present invention provides a retrieval method for a second knowledge graph, which converts a question input by a user into a word vector representation through a word vector model, and calculates the cosine similarity between the word vector representation of the question input by the user and the word vector representation of the first label corresponding to each entity in the second knowledge graph, and retains entities whose cosine similarity is greater than or equal to a cosine similarity threshold, and then calculates the cosine similarity between the word vector representation of the question input by the user and the word vector representation of the second label corresponding to the retained entity, and sorts them in order from large to small, and obtains the first U multimodal data that establishes an association relationship with the entity and returns them to the user, wherein the cosine similarity threshold is a custom parameter, and preferably, the cosine similarity threshold is set to 0.8.

[0039] The above describes an embodiment of the present embodiment, but the present embodiment is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present embodiment, ordinary technicians in this field can also make many forms, all of which are within the protection of the present embodiment.

Claims

1. A data association method, It is characterized in that The following steps are involved: Step S101, extracting labels of M multimodal data; Multimodal data includes: text data, image data and video data; Labels include: primary labels and secondary labels; The first-level labels are represented by words, and the second-level labels are represented by text summaries; Step S102, constructing a first knowledge graph according to the labels of the M multimodal data; The first knowledge graph includes: M entities, initial features of entities and connection relationships between entities; Each entity is assigned a unique code; The M entities are associated with the unique codes of the M multimodal data respectively; The initial features of an entity are represented by its label; The connection relationship between entities includes: if the similarity of the initial features corresponding to any two entities is greater than or equal to the similarity threshold, then there is a connection relationship between the two entities; Step S103, inputting the first knowledge graph into the error correction model, and outputting a first adjacency matrix; The element value of the jth column of the i-th row of the first adjacency matrix indicates whether there is a connection relationship between the i-th entity and the j-th entity of the first knowledge graph. If there is a connection relationship, the element value of the jth column of the i-th row of the first adjacency matrix is ​​assigned to 1, otherwise it is assigned to 0, where 1≤i≤M, 1≤j≤M; Step S104, correcting the first knowledge graph according to the first adjacency matrix to obtain a second knowledge graph; The representation of the second knowledge graph is the same as that of the first knowledge graph.

2. A data association method according to claim 1, It is characterized in that M, primary label, and similarity threshold are all custom parameters.

3. A data association method according to claim 1, It is characterized in that Extracting labels for multimodal data includes the following steps: Step S201: for text data, firstly, the text data is segmented by a word segmenter, then the segmented words are converted into word vector representations by a word vector model, and finally the word vector representations of multiple segmented words are input into a semantic analysis model to output a primary label and a secondary label; Step S202: for the image data, obtaining the primary label and the secondary label of the image data through the image-to-text model; Step S203: for the video data, firstly perform frame processing on the video data at equal time intervals to obtain A frame images, then convert the A frame images into grayscale histogram representation, and calculate the Bhattacharyya distance of adjacent grayscale histograms, and if it is determined that the Bhattacharyya distance is greater than or equal to a preset distance threshold, then use the grayscale image of the next frame as a key frame image, until an image sequence consisting of B key frame images is obtained, and finally input the image sequence into a time series analysis model to obtain the primary label and secondary label of the video data, wherein A and B are both custom parameters; The grayscale histogram is represented by a vector with 256 dimensions, and the value of each dimension represents the number of pixels with the same grayscale level as the dimension index; The calculation formula of the Bhattacharyya distance of adjacent grayscale histograms is as follows: ; Where 1≤a≤A, Represents the Bhattacharyya distance between the a-th grayscale histogram and the a-1-th grayscale histogram, and Represent the values ​​of the b-th dimension of the a-th and a-1-th grayscale histograms respectively.

4. A data association method according to claim 1, It is characterized in that The calculation formula for the similarity Sim of the initial features corresponding to any two entities is as follows: ; Where 1≤i≤M, 1≤j≤M, i≠j, and Represent the word vector representation of the first-level label of the i-th entity and the j-th entity, and Represent the word vector representation of the secondary label of the i-th entity and the j-th entity respectively, represents the Euclidean distance between the word vector representations of the first-level labels of the i-th entity and the j-th entity, The Manhattan distance between the word vector representations of the secondary labels of the i-th entity and the j-th entity, and Both are custom parameters, and their sum is 1.

5. A data association method according to claim 1, It is characterized in that The error correction model includes: conversion layer, splicing layer, padding layer, and hidden layer; The conversion layer is used to convert the primary label and secondary label corresponding to the initial features of each entity into word vector representation through the word vector model; The concatenation layer is used to concatenate the word vector representations corresponding to the primary label and the secondary label of each entity to obtain a combined vector; The padding layer is used to unify the number of dimensions of the combined vector of each entity to the maximum number of dimensions of the combined vector of the entity. For word vectors whose number of dimensions is less than the maximum number of dimensions, the average value of the remaining dimensions is taken to fill them. The hidden layer inputs the first knowledge graph and outputs the first adjacency matrix.

6. A data association method according to claim 5, It is characterized in that The calculation formula for the hidden layer includes: ; ; ; Where 1≤u≤M, S represents the first adjacency matrix of the hidden layer output, represents the set of entities that have a connection relationship with the u-th entity, and Respectively represent the combination vectors of the u-th entity and the v-th entity, represents the update vector of the u-th entity, and Respectively represent the weight parameter and distance coefficient between the u-th entity and the v-th entity, represents the Manhattan distance between the u-th entity and the v-th entity, represents the weight parameter corresponding to the u-th entity, and denote the first weight parameter and the second weight parameter respectively, represents the bias parameter, It indicates that the update vectors of M entities are stacked, T indicates the transpose operation, exp indicates the power operation of a natural constant, and sigmoid indicates the sigmoid activation function.

7. A data association method according to claim 1, It is characterized in that According to step S101 to step S102, a first knowledge graph is obtained as training data of training samples for training an error correction model, and the connection relationship of entities in the first knowledge graph is corrected by manual labeling to obtain a second adjacency matrix, the second adjacency matrix has the same representation as the first adjacency matrix, the second adjacency matrix serves as sample labels of training samples for training the error correction model, and the difference between the second adjacency matrix and the first adjacency matrix is ​​designated as the loss function for training the error correction model.

8. A data association method according to claim 1, It is characterized in that When the element value in the first adjacency matrix is ​​0, the connection relationship between the entities corresponding to the first knowledge graph is deleted. When the element value in the first adjacency matrix is ​​1, the connection relationship between the entities corresponding to the first knowledge graph is established until all element values ​​in the first adjacency matrix are traversed to obtain the second knowledge graph.

9. A data association device, It is characterized in that Executing a data association method as claimed in any one of claims 1 to 8, comprising: A label extraction module, which is used to extract labels of M multimodal data; A first knowledge graph construction module, which is used to construct a first knowledge graph according to the labels of M multimodal data; A first knowledge graph error correction module, which is used to input the first knowledge graph into the error correction model and output a first adjacency matrix; The second knowledge graph generation module is used to correct the first knowledge graph according to the first adjacency matrix to obtain the second knowledge graph.

10. A readable storage medium, It is characterized in that It stores non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by a computer, a data association method according to any one of claims 1 to 8 is executed.

Citation Information

Patent Citations

  • Intelligent dialogue management system and method based on AI

    CN117591657A

  • Power grid dispatching multi-mode knowledge graph construction method and system

    CN118035463A

  • System And Method For A Unified Semantic Ranking of Compositions of Ontological Subjects And The Applications Thereof

    US20100293166A1

Cited By

  • Security human-vehicle multi-mode retrieval method and device, electronic equipment and storage medium

    CN120234439A