Knowledge graph construction method and device, electronic equipment and storage medium

Through the bidirectional cross-modal attention mechanism, the multimodal data in fan equipment maintenance is converted into vectors and fused to build a knowledge graph, which solves the problem of unused multi-source heterogeneous data, and significantly improves the richness and accuracy of the knowledge graph. sex.

CN118966338APending Publication Date: 2024-11-15NANJING HUADUN ELECTRIC POWER INFORMATION SAFETY EVALUATION CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411019166.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

Multi-source heterogeneous data generated during daily maintenance and inspection of fan equipment, especially text, image and video data, have not been fully utilized, resulting in insufficient richness and accuracy of the knowledge graph.

Method used

A two-way cross-modal attention mechanism is used to convert text, image and video data into corresponding vectors, and a knowledge graph is constructed by fusing these vectors to realize the association and fusion of multimodal data.

Benefits of technology

The richness and accuracy of the knowledge graph are significantly improved. Through the integration and complementation of multimodal data, more comprehensive and rich semantic information is captured, solving the problems of redundancy and information unutilization in a single data mode.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118966338A_ABST
    Figure CN118966338A_ABST
Patent Text Reader

Abstract

The invention discloses a knowledge graph construction method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining text data, image data and video data; converting the text data into a text vector, converting the image data into an image vector, and converting the video data into a video vector; based on a bidirectional cross-modal attention mechanism, determining a target image vector and a target video vector matched with the target text vector; and determining a fusion vector according to the target text vector, the target image vector and the target video vector, and performing knowledge graph construction according to the fusion vector. According to the method, association and fusion of multi-modal data can be realized, and the richness and accuracy of the knowledge graph are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a knowledge graph construction method, device, electronic device and storage medium. Background Art

[0002] With the development of artificial intelligence and knowledge graph technology, the management and application of business data in various industries have become more systematic and intelligent. For example, for the wind power business of the new energy power industry, historical data related to the maintenance and repair of wind turbine equipment can be obtained to extract entities and extract the relationships between entities, build a knowledge graph of entities and corresponding relationships, and update the knowledge graph based on the new data after obtaining new data.

[0003] However, the daily maintenance and inspection of wind turbine equipment will generate massive, multi-source data, such as text data such as equipment fault repair records, equipment operation status images generated by online monitoring systems, equipment maintenance videos and other data. Power operation and inspection data is gradually showing characteristics such as heterogeneity, diversification, complexity and isolation. The common knowledge graph construction method is still used, and the internal relationship between multiple modal data has not been deeply explored, and the potential information has not been fully utilized. Therefore, in order to improve the richness and quality of the knowledge graph, a knowledge graph construction method based on multimodal data is urgently needed. Summary of the invention

[0004] The present invention provides a knowledge graph construction method, device, electronic device and storage medium to achieve the association and fusion of multimodal data, significantly improving the richness and accuracy of the knowledge graph.

[0005] In a first aspect, an embodiment of the present invention provides a method for constructing a knowledge graph, the method comprising:

[0006] Acquire text data, image data, and video data;

[0007] Converting the text data into text vectors, converting the image data into image vectors, and converting the video data into video vectors;

[0008] Based on the bidirectional cross-modal attention mechanism, the target image vector and the target video vector matching the target text vector are determined;

[0009] A fusion vector is determined according to the target text vector, the target image vector and the target video vector, and a knowledge graph is constructed according to the fusion vector.

[0010] In a second aspect, an embodiment of the present invention further provides a knowledge graph construction device, the device comprising:

[0011] A data acquisition module, used to acquire text data, image data and video data;

[0012] A vector conversion module, used for converting the text data into a text vector, converting the image data into an image vector, and converting the video data into a video vector;

[0013] A vector determination module, used to determine a target image vector and a target video vector matching the target text vector based on a bidirectional cross-modal attention mechanism;

[0014] The vector fusion module is used to determine a fusion vector according to a target text vector, a target image vector and a target video vector, and to construct a knowledge graph according to the fusion vector.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a knowledge graph construction method as described in any one of the embodiments of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a storage medium storing computer executable instructions, which, when executed by a computer processor, are used to execute a knowledge graph construction method as described in any one of the embodiments of the present invention.

[0017] The technical solution of the embodiment of the present invention obtains text data, converts the text data into text vectors, obtains image data, converts the image data into image vectors, obtains video data, converts the video data into video vectors, determines the target image vector and target video vector that match the target text vector based on bidirectional cross-modal attention, determines the fusion vector based on the target text vector, the target image vector and the target video vector, and constructs the knowledge graph based on the fusion vector. The technical solution of this embodiment solves the problem of single knowledge graph in the prior art, realizes the association and fusion of multimodal data, and significantly improves the richness and accuracy of the knowledge graph.

[0018] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 It is a flowchart of a knowledge graph construction method provided in the first embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of multimodal data fusion provided in Embodiment 1 of the present invention;

[0022] Figure 3 It is a flow chart of a knowledge graph construction method provided in the second embodiment of the present invention;

[0023] Figure 4 It is a schematic diagram of the structure of a knowledge graph construction device provided in Embodiment 3 of the present invention;

[0024] Figure 5 It is a structural schematic diagram of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the embodiments of the present application, some existing solutions in the industry such as certain software, components, models, etc. may be mentioned, and they should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or must use the solution.

[0027] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0028] Embodiment 1

[0029] Figure 1 A flowchart of a knowledge graph construction method is provided for the first embodiment of the present invention. This embodiment is applicable to the situation where a knowledge graph is constructed after fusing multimodal data. The method can be executed by a knowledge graph construction device. The knowledge graph construction device can be implemented in the form of hardware and / or software. The knowledge graph construction device can be configured in a server.

[0030] like Figure 1 As shown, the method includes:

[0031] S110, acquiring text data, image data, and video data.

[0032] Among them, text data, image data and video data can be both historical data required in the process of building a knowledge graph and new data for updating the knowledge graph after the knowledge graph is built. This embodiment does not limit this.

[0033] Specifically, taking the maintenance and inspection business scenario of wind turbine equipment in renewable energy power as an example, the business data includes structured data, semi-structured data and unstructured data. Structured data includes real-time and historical operation data of wind turbines collected by each wind farm, wind turbine information data, etc. Semi-structured data includes wind turbine maintenance log data stored in JSON or XML files, etc. Unstructured data mainly includes long text type data and other modal data such as images and videos. Among them, long text data mainly includes wind turbine operation procedures, maintenance procedures, test reports, etc., and image and video data mainly come from the collection of online monitoring systems. Therefore, the data required to build or update the knowledge graph can be divided into text data, image data and video data.

[0034] Furthermore, data preprocessing may be performed on each data, such as data cleaning processing, data noise reduction and dimensionality reduction processing, etc. This embodiment does not limit the specific method used for data preprocessing.

[0035] In this embodiment, newly added image data and video data are added when constructing and updating the knowledge graph, which can provide rich background knowledge to enrich the representation of entities and concepts in the knowledge graph, effectively solve the long-tail relationship problem caused by single text data, and improve the richness of the knowledge graph.

[0036] Furthermore, after acquiring text data, image data, and video data, entity extraction and relationship extraction are performed based on each data. This embodiment does not limit the specific method used in the extraction process. After extraction, the <head entity, relationship, tail entity> triple is obtained to complete the relationship construction of the knowledge graph. However, the above process only realizes the relationship mapping between different types of data and entities. The association relationship between text data, image data, and video data is not mapped, and there is redundancy in the knowledge graph. Therefore, in this embodiment, it is necessary to perform entity disambiguation, entity alignment and other operations on multi-source heterogeneous data, integrate data of different modalities into the same network, realize multi-dimensional fusion of multimodal data, and eliminate redundant information.

[0037] S120, converting the text data into a text vector, converting the image data into an image vector, and converting the video data into a video vector.

[0038] In this embodiment, for different types of data, corresponding data processing methods can be adopted to perform feature extraction and vector conversion.

[0039] Specifically, converting text data into text vectors can be achieved through a pre-trained language model, such as a BERT (Bidirectional Encoder Representations from Transformers) model, an ELMO (Embeddings from Language Models), etc. This embodiment does not limit the type of language model.

[0040] Furthermore, converting the text data into a text vector may further include:

[0041] Performing stop word removal processing on the text data, and performing stem extraction and morphological restoration processing on the processed text data;

[0042] The processed text data is divided into word units, and each word unit is converted into a word unit vector, and the position code of each word unit is determined;

[0043] Each word element vector and position code are concatenated to obtain a text vector.

[0044] Stop words refer to words that contribute little to the meaning of the text and are ineffective in text analysis. Removing stop words from the text can reduce the data dimension, reduce the amount of calculation and storage space, and highlight the key information in the text data.

[0045] Stemming refers to removing the affixes of a word and extracting the main part of the word. Stemming can reduce the diversity of vocabulary, reduce the dimension of text data, and improve the efficiency of text data processing.

[0046] Lemmatization is the process of converting a word into its basic form in the dictionary to obtain the correct word form. Lemmatization can better preserve the original semantics of a word and improve the accuracy of text processing.

[0047] This embodiment does not limit the specific methods used for removing stop words, extracting stems, and restoring morphological forms.

[0048] Token refers to the basic unit obtained after word segmentation of text data, which can be a word, number, punctuation mark, etc. The processed text data can be divided into tokens by word segmentation. Each token obtained after word segmentation is matched with the dictionary, and the token vector corresponding to each token can be obtained. Position encoding is used to indicate the position of a token in a sequence.

[0049] In this embodiment, the processed text data is divided into word units, which can be achieved through the RoBERTa (Robustly Optimized BERT Approach) model. The RoBERTa model uses a dynamic mask, and a different mask is used each time the text data is input, which increases the data diversity. At the same time, the optimization function is improved. When encoding the text data, the BPE (Byte-Pair Encoding) method is used. Specifically, the text data is segmented through the word segmenter of the RoBERTa model, and the RoBERTa model generates a word unit vector for each word unit, and generates a position code corresponding to the word unit.

[0050] Each word element vector and its position code are concatenated to obtain the text vector.

[0051] Furthermore, converting the image data into an image vector may include:

[0052] Dividing the image data into image blocks, flattening each image block, and determining a position code of each image block;

[0053] Generate an image block sequence according to each image block after flattening and the position code of each image block;

[0054] According to the residual network, the image patch sequence is converted into an image vector.

[0055] Before processing the image data, the image data may be preprocessed. The preprocessing may include unifying the image size, cropping, scaling, etc., so that each image has a uniform size in the subsequent processing process.

[0056] When slicing an image, the size and step length of the slices can be determined in advance. The slice size is usually a fixed size, such as 8×8 pixels, 16×16 pixels, etc. The step length determines the degree of overlap between two adjacent slices. When the step length is the slice size, there is no overlap, and when the step length is less than the slice size, there is overlap. When the image data is sliced, in the subsequent feature extraction, local features of the image blocks can be extracted first, and then the features of each image block can be integrated for global analysis, so as to realize multi-scale analysis of image data and increase the diversity of image data.

[0057] Image block flattening refers to converting image blocks from two-dimensional or multi-dimensional matrix form to one-dimensional vector form. After flattening, the obtained image block vector can be used as the input of the fully connected layer.

[0058] The position encoding of the image block refers to the sequence number of the image block in the sequence. The image block sequence constructed by the flattened image blocks and the position encoding of each image block is used as the input of the residual network (ResNet).

[0059] The image block sequence is input into the residual network to obtain the image vector output by the residual network. Specifically, the image block sequence is feature extracted through a series of convolution kernels of the residual network, and then the image block sequence is converted into a feature image through the residual block, and finally the feature image is converted into a feature vector through the global average pooling layer as the image vector matching the input image block sequence.

[0060] Furthermore, converting the video data into a video vector may further include:

[0061] Dividing the video data into video segments, each image in the video segment is a continuous video frame image;

[0062] According to the 3D convolutional neural network, each video clip is converted into a video vector.

[0063] Before the video data is segmented, different video data can be converted into the same reading format. At the same time, each frame of the video data is subjected to image preprocessing. Image preprocessing can include adjusting the image size, normalizing pixel values, and trimming edges. In order to enhance the generalization ability of the three-dimensional convolutional neural network, image enhancement operations such as random rotation, flipping, and brightness adjustment can also be performed on each frame of the image.

[0064] In this embodiment, the video data can be divided into video segments according to the preset number of video segment images. Exemplarily, the number of video segment images included in each video segment can be 16. And each frame image in a video segment should be a continuous video frame image. Specifically, the video segment can be divided by a sliding window sampling method, and the window length is set to 16.

[0065] By segmenting the video data and ensuring that the video segments are continuous video frame images, it is possible to capture the motion information in the video data while capturing local features, and extract more accurate video semantic features. At the same time, segmenting the video data into video segments of uniform size reduces the computational complexity, improves the adaptability to videos of different lengths, and improves the generalization ability of the 3D convolutional neural network model.

[0066] Three-dimensional convolutional neural network, also known as 3D CNN (Convolutional Neural Network), can be used to process three-dimensional data. The convolution kernel slides and calculates in three dimensions (height, width, depth), which can capture the discriminative features of spatial and temporal dimensions. The 3D CNN architecture can generate multiple information channels based on adjacent video frame images, and perform convolution and downsampling in each channel separately, and obtain the final video feature representation by combining information from video channels.

[0067] S130. Based on the bidirectional cross-modal attention mechanism, determine the target image vector and the target video vector that match the target text vector.

[0068] Among them, the bidirectional cross-modal attention mechanism can be used for multimodal data processing. The bidirectional cross-modal attention mechanism can fuse the information of multimodal data and capture the semantic associations between them by calculating the similarities between different modal data. Through the bidirectional cross-modal attention mechanism, this embodiment can simultaneously focus on the key information between the three modalities of text, image and video, and generate a more accurate cross-modal representation, that is, a fusion feature containing better modal association information.

[0069] In this embodiment, the semantic association between different modal information is achieved through a bidirectional cross-modal attention mechanism, and the text vector can be matched with the entities in the image vector and the video vector, so that the knowledge graph can achieve more accurate retrieval and reasoning.

[0070] Furthermore, S130 may include:

[0071] Calculating a first target text feature vector based on the similarity between the target text vector and each image vector and each image vector, and calculating a target image feature vector based on the similarity between the target image vector and each text vector and each text vector;

[0072] Determining a target image vector matching the target text vector from among the image vectors according to the first target text feature vector and the image feature vectors;

[0073] Calculating a second target text feature vector based on the similarity between the target text vector and each video vector and each video vector, and calculating a target video feature vector based on the similarity between the target video vector and each text vector and each text vector;

[0074] According to the second target text feature vector and each video feature vector, a target video vector matching the target text vector is determined in each video vector.

[0075] Specifically, calculate the target text vector t and each image vector i n The similarity between them is used as the attention score a mn , and normalize each attention score to get α mn According to the target text vector t and each image vector i n The attention score α between mn , and each image vector i n Perform weighted summation to obtain the first target text feature vector T corresponding to the target text vector m1 Specifically, it can be calculated by the following formula:

[0076] Calculate the target image vector i and each text vector t m The similarity between them is used as the attention score b nm For each attention score b nm After normalization, we get β nm According to the target image vector i and each text vector t m The attention score β between nm And each text vector t m , get the target image feature vector corresponding to the target image vector. Specifically, it can be calculated by the following formula:

[0077]

[0078] According to the first target text feature vector and each image feature vector, a target image vector matching the target text vector is determined in each image vector. Specifically, the similarity between the first target text feature vector and each image feature vector can be calculated, and according to the similarity, a target image vector matching the target text vector is determined in each image vector. Exemplarily, an image vector whose similarity between an image feature vector and the first target text feature vector is greater than or equal to a preset similarity threshold can be used as a target image vector matching the target text vector. It is also possible to sort each image vector in descending order according to the similarity between the image feature vector and the first target text feature vector, and select the first preset number of image vectors as target image vectors matching the target text vector.

[0079] Similarly, calculate the target text vector t and each video vector v t The similarity between them is taken as the attention score d mt , and normalize each attention score to get δ mt According to the target text vector t and each video vector v t The attention score δ between mt , and each video vector v t , and obtain the second target text feature vector T2 corresponding to the target text vector t. Specifically, it can be calculated by the following formula:

[0080] Calculate the target video vector v and each text vector t m The similarity between them is taken as the attention score s tm , and normalize each attention score to get γ tm According to the target video vector v and each text vector t m The attention score between tm , and each text vector t m , get the target video feature vector corresponding to the target video vector v. Specifically, it can be calculated by the following formula:

[0081]

[0082] Similarly, based on the second target text feature vector and each video feature vector, a target video vector matching the target text vector is determined in each video vector, and the similarity between the second target text feature vector and each video feature vector can be calculated. Based on the similarity, a target video vector matching the target text vector is determined in each video vector. Exemplarily, a video vector whose similarity between a video feature vector and the second target text feature vector is greater than or equal to a preset similarity threshold can be used as a target video vector matching the target text vector. It is also possible to sort each video vector in descending order according to the similarity between the video feature vector and the first target text feature vector, and select the first preset number of video vectors as target video vectors matching the target text vector.

[0083] S140. Determine a fusion vector according to the target text vector, the target image vector, and the target video vector, and construct a knowledge graph according to the fusion vector.

[0084] In this embodiment, the image vector and video vector have richer visual information, and the text data can provide more accurate and detailed semantic information. Linking the image vector and video vector with the text vector, and fusing the text vector, image vector and video vector corresponding to the same entity to form a unified fusion vector, can give full play to the advantages of each modal data and achieve information complementarity. By constructing a knowledge graph through fusion vectors, the semantic information of entities and relationships can be further captured, so that the knowledge graph can obtain more comprehensive and rich semantic information, realize the integration and complementarity of multimodal data, realize modal complementarity and knowledge disambiguation, and improve the richness and quality of the knowledge graph.

[0085] Figure 2 A schematic diagram of multimodal data fusion is provided, such as Figure 2 As shown in the figure, the text data is converted into text vectors based on the RoBERTa model, the image data is converted into image vectors based on the residual network, and the video data is converted into video vectors based on the 3D CNN. Based on the bidirectional cross-modal attention mechanism, the similarity between the image vector and the video vector and the text vector is calculated, and the image vector and the video vector are linked to the text vector, and finally the text vector, image vector and video vector corresponding to the same entity are obtained.

[0086] Furthermore, S140 may include:

[0087] Aggregating at least one target image vector that matches the target text vector to obtain an aggregated image vector, and aggregating at least one target video vector that matches the target text vector to obtain an aggregated video vector;

[0088] The aggregated image vector and the aggregated video vector are fused with the aggregated video vector to obtain a fused vector.

[0089] In this embodiment, the image vector and the video vector can be aggregated separately and then fused with the text vector. On the one hand, aggregating the image vector and the video vector separately first can reduce the data dimension and complexity, and reduce the amount of calculation and processing difficulty when fused with the text vector later. On the other hand, the aggregation operation helps to extract the main features and common patterns in the respective modalities of the image and video, making these features more significant and representative when fused with the text. At the same time, separate aggregation can reduce the differences between the three modalities of image, video and text to a certain extent, making them more coordinated and easier to handle when fused.

[0090] The technical solution of the embodiment of the present invention obtains text data, converts the text data into text vectors, obtains image data, converts the image data into image vectors, obtains video data, converts the video data into video vectors, determines the target image vector and target video vector that match the target text vector based on bidirectional cross-modal attention, determines the fusion vector based on the target text vector, the target image vector and the target video vector, and constructs the knowledge graph based on the fusion vector. The technical solution of this embodiment solves the problem of single knowledge graph in the prior art, realizes the association and fusion of multimodal data, and significantly improves the richness and accuracy of the knowledge graph.

[0091] Embodiment 2

[0092] Figure 3 A flowchart of a knowledge graph construction method provided in the second embodiment of the present invention. Based on the above embodiment, the embodiment of the present invention further specifies the process of image vector conversion.

[0093] like Figure 3 As shown, the method includes:

[0094] S210: Acquire text data, image data, and video data.

[0095] S220: Convert the text data into a text vector.

[0096] The above embodiments have described the process of acquiring data in three modes, namely text, image and video, and performing vector conversion respectively, which will not be described in detail in this embodiment.

[0097] S230: Divide the image data into image blocks, flatten each image block, and determine a position code for each image block.

[0098] S240 , generating an image block sequence according to each image block after flattening and the position code of each image block.

[0099] The above embodiments have described the specific process of image data slicing, flattening, and generating an image block sequence during image vector conversion, which will not be repeated in this embodiment.

[0100] S250, converting the image block sequence into an image vector according to a neural network structure search algorithm.

[0101] Based on the above embodiment, this embodiment provides an improved residual network structure to illustrate the process of image vector conversion. Specifically, on the basis of the residual network, a neural network structure search algorithm and an adaptive mechanism are introduced, so that the residual network has a dynamic network structure, so that the neural network structure can be dynamically adjusted according to the characteristics of the input image data, further improving the feature extraction performance and generalization ability of the residual network.

[0102] Furthermore, S250 may include:

[0103] Determine a search space for a neural network structure, the search space comprising at least two numbers of network layers, at least two layer parameters, at least two layer connection modes, and at least two operation types;

[0104] Determine, according to a neural network structure search algorithm, a target neural network structure that matches the image block sequence, as well as the number of network layers, layer parameters, connection mode, and operation type of the target neural network structure;

[0105] Based on the target neural network structure, the image patch sequence is converted into an image vector.

[0106] The search space refers to the set of all possible neural network structures, which defines the scope and possibility of neural network structures that can be explored and selected by the search algorithm. The search space can include the following aspects: the number of network layers, such as the number of convolutional layers, fully connected layers, and recurrent layers; layer parameters, such as the convolution kernel size, step size, number of channels, and number of neurons in the fully connected layer; layer connection methods, such as sequential connections, skip connections, etc.; operation types, such as different types of convolution operations (such as ordinary convolution, depthwise separable convolution, etc.), pooling operations (such as maximum pooling, average pooling, etc.), etc.

[0107] Furthermore, in this embodiment, residual block connection modes can be defined, including basic connection (BasicBlock), bottleneck connection (Bottleneck Block), residual fully connected neural network (Residual Fully Connected NeuralNetwork, RFCN) and randomly connected residual fully connected network (Randomly Connected Residual FullyConnected Network, RRFCN). The number of network layers can be defined from shallow ResNet18 to deep ResNet152.

[0108] The neural network structure search algorithm may include random search, Bayesian optimization search, reinforcement learning, evolutionary algorithm, gradient descent algorithm, etc. This embodiment does not limit the selected neural network structure search algorithm.

[0109] In this embodiment, after the search algorithm is determined, the search algorithm is used to find the optimal target neural network structure in the search space according to the complexity of the input image block sequence, and the number of network layers, layer parameters, connection methods, operation types, etc. of the target neural network structure are dynamically adjusted until the target neural network structure with the best effect is obtained.

[0110] The target neural network structure is adopted to convert the image block sequence and its image block encoding into image vectors.

[0111] Furthermore, in this embodiment, each image data can be subjected to a neural network structure search algorithm to determine the corresponding neural network structure, and then the corresponding image vector conversion can be performed. The image block sequence can also be classified according to its complexity, and different types can be determined by using a neural network structure search algorithm to determine different neural network structures. When performing vector conversion on new image data, the classification corresponding to the complexity of its image block sequence is determined, and the corresponding neural network structure is determined, and the corresponding image vector conversion is performed using its neural network structure.

[0112] S260, converting the video data into a video vector.

[0113] S270. Based on the bidirectional cross-modal attention mechanism, determine the target image vector and the target video vector that match the target text vector.

[0114] S280. Determine a fusion vector according to the target text vector, the target image vector, and the target video vector, and construct a knowledge graph according to the fusion vector.

[0115] The processing process of the bidirectional cross-modal attention mechanism and the specific process of vector fusion have been described in the above embodiment, and will not be repeated in this embodiment.

[0116] The technical solution of the embodiment of the present invention is to obtain text data, convert the text data into text vectors, and obtain image data. The image data is converted into image vectors using the improved residual network after introducing the neural network structure search algorithm, so that the residual network has a dynamic network structure, thereby dynamically adjusting the network architecture according to the characteristics of the input image data, and further improving the feature extraction performance and generalization ability of the residual network. Video data is obtained, the video data is converted into video vectors, and based on bidirectional cross-modal attention, the target image vector and target video vector matching the target text vector are determined to achieve semantic associations between different modal information, and the text vector can be corresponded to entities in the image vector and the video vector, so that the knowledge graph can achieve more accurate retrieval and reasoning. According to the target text vector, the target image vector and the target video vector, the fusion vector is determined, and the knowledge graph is constructed based on the fusion vector, which can further capture the semantic information of entities and relationships, so that the knowledge graph can obtain more comprehensive and rich semantic information, realize the association, integration and complementarity of multimodal data, realize modal complementarity and knowledge disambiguation, and improve the richness and quality of the knowledge graph.

[0117] Embodiment 3

[0118] Figure 4 A schematic diagram of the structure of a knowledge graph construction device provided in Example 3 of the present invention.

[0119] like Figure 4 As shown, the device comprises:

[0120] The data acquisition module 310 is used to acquire text data, image data and video data;

[0121] A vector conversion module 320, configured to convert the text data into a text vector, convert the image data into an image vector, and convert the video data into a video vector;

[0122] A vector determination module 330, configured to determine a target image vector and a target video vector matching the target text vector based on a bidirectional cross-modal attention mechanism;

[0123] The vector fusion module 340 is used to determine a fusion vector according to the target text vector, the target image vector and the target video vector, and to construct a knowledge graph according to the fusion vector.

[0124] The technical solution of the embodiment of the present invention obtains text data, converts the text data into text vectors, obtains image data, converts the image data into image vectors, obtains video data, converts the video data into video vectors, determines the target image vector and target video vector that match the target text vector based on bidirectional cross-modal attention, determines the fusion vector based on the target text vector, the target image vector and the target video vector, and constructs the knowledge graph based on the fusion vector. The technical solution of this embodiment solves the problem of single knowledge graph in the prior art, realizes the association and fusion of multimodal data, and significantly improves the richness and accuracy of the knowledge graph.

[0125] Based on the above embodiment, the vector conversion module 320 includes:

[0126] A text data processing unit, used to remove stop words from the text data, and perform stem extraction and morphological restoration on the processed text data;

[0127] A text data segmentation unit, used to segment the processed text data into word units, convert each word unit into a word unit vector, and determine the position code of each word unit;

[0128] The text vector determination unit is used to concatenate each word element vector and position code to obtain a text vector.

[0129] Based on the above embodiment, the vector conversion module 320 includes:

[0130] An image data segmentation unit, used for segmenting the image data into image blocks, flattening each image block, and determining a position code of each image block;

[0131] An image block sequence determining unit, configured to generate an image block sequence according to each image block after flattening and a position code of each image block;

[0132] The image vector determination unit is used to convert the image block sequence into an image vector according to a neural network structure search algorithm.

[0133] Based on the above embodiment, the image vector determination unit is specifically used for:

[0134] Determine a search space for a neural network structure, the search space comprising at least two numbers of network layers, at least two layer parameters, at least two layer connection modes, and at least two operation types;

[0135] Determine, according to a neural network structure search algorithm, a target neural network structure that matches the image block sequence, as well as the number of network layers, layer parameters, connection mode, and operation type of the target neural network structure;

[0136] Based on the target neural network structure, the image patch sequence is converted into an image vector.

[0137] Based on the above embodiment, the vector conversion module 320 includes:

[0138] A video data segmentation unit, used for segmenting the video data into video segments, wherein each image in the video segment is a continuous video frame image;

[0139] The video vector determination unit is used to convert each video clip into a video vector according to a three-dimensional convolutional neural network.

[0140] Based on the above embodiment, the vector determination module 330 includes:

[0141] A first feature vector determination unit, configured to calculate a first target text feature vector based on the similarity between the target text vector and each image vector and each image vector, and to calculate a target image feature vector based on the similarity between the target image vector and each text vector and each text vector;

[0142] A target image vector determining unit, configured to determine a target image vector matching the target text vector from among the image vectors based on the first target text feature vector and the image feature vectors;

[0143] A second feature vector determination unit, configured to calculate a second target text feature vector based on the similarity between the target text vector and each video vector and each video vector, and to calculate a target video feature vector based on the similarity between the target video vector and each text vector and each text vector;

[0144] The target video vector determining unit is used to determine a target video vector matching the target text vector in each video vector according to the second target text feature vector and each video feature vector.

[0145] Based on the above embodiment, the vector fusion module 340 includes:

[0146] a vector aggregation unit, configured to aggregate at least one target image vector that matches the target text vector to obtain an aggregated image vector, and to aggregate at least one target video vector that matches the target text vector to obtain an aggregated video vector;

[0147] The vector fusion unit is used to fuse the aggregated image vector and the aggregated video vector with the aggregated video vector to obtain a fused vector.

[0148] The knowledge graph construction device provided in the embodiment of the present invention can execute the knowledge graph construction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] Embodiment 4

[0150] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0151] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0152] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0153] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the knowledge graph construction method.

[0154] In some embodiments, the knowledge graph construction method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the knowledge graph construction method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the knowledge graph construction method in any other appropriate manner (e.g., by means of firmware).

[0155] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0156] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0157] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0158] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0159] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0160] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0161] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0162] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A knowledge graph construction method, characterized in that: include: Acquire text data, image data, and video data; Converting the text data into text vectors, converting the image data into image vectors, and converting the video data into video vectors; Based on the bidirectional cross-modal attention mechanism, the target image vector and the target video vector matching the target text vector are determined; A fusion vector is determined according to the target text vector, the target image vector and the target video vector, and a knowledge graph is constructed according to the fusion vector.

2. The method according to claim 1, characterized in that Converting the text data into a text vector includes: Performing stop word removal processing on the text data, and performing stem extraction and morphological restoration processing on the processed text data; The processed text data is divided into word units, and each word unit is converted into a word unit vector, and the position code of each word unit is determined; Each word element vector and position code are concatenated to obtain a text vector.

3. The method according to claim 1, characterized in that Converting the image data into an image vector includes: Dividing the image data into image blocks, flattening each image block, and determining a position code of each image block; Generate an image block sequence according to each image block after flattening and the position code of each image block; According to the neural network structure search algorithm, the image block sequence is converted into an image vector.

4. The method according to claim 3, characterized in that According to the neural network structure search algorithm, the image block sequence is converted into an image vector, including: Determine a search space for a neural network structure, the search space comprising at least two numbers of network layers, at least two layer parameters, at least two layer connection modes, and at least two operation types; Determine, according to a neural network structure search algorithm, a target neural network structure that matches the image block sequence, as well as the number of network layers, layer parameters, connection mode, and operation type of the target neural network structure; Based on the target neural network structure, the image patch sequence is converted into an image vector.

5. The method according to claim 1, characterized in that: Converting the video data into a video vector includes: Dividing the video data into video segments, each image in the video segment is a continuous video frame image; According to the 3D convolutional neural network, each video clip is converted into a video vector.

6. The method according to claim 1, characterized in that Based on the bidirectional cross-modal attention mechanism, the target image vector matching the target text vector is determined, including: Calculating a first target text feature vector based on the similarity between the target text vector and each image vector and each image vector, and calculating a target image feature vector based on the similarity between the target image vector and each text vector and each text vector; Determining a target image vector matching the target text vector from among the image vectors according to the first target text feature vector and the image feature vectors; Based on the bidirectional cross-modal attention mechanism, the target video vector matching the target text vector is determined, including: Calculating a second target text feature vector based on the similarity between the target text vector and each video vector and each video vector, and calculating a target video feature vector based on the similarity between the target video vector and each text vector and each text vector; According to the second target text feature vector and each video feature vector, a target video vector matching the target text vector is determined in each video vector.

7. The method according to claim 1, characterized in that According to the target text vector, the target image vector and the target video vector, a fusion vector is determined, including: Aggregating at least one target image vector that matches the target text vector to obtain an aggregated image vector, and aggregating at least one target video vector that matches the target text vector to obtain an aggregated video vector; The aggregated image vector and the aggregated video vector are fused with the aggregated video vector to obtain a fused vector.

8. A knowledge graph construction device, characterized in that: include: A data acquisition module, used to acquire text data, image data and video data; A vector conversion module, used for converting the text data into a text vector, converting the image data into an image vector, and converting the video data into a video vector; A vector determination module, used to determine a target image vector and a target video vector matching a target text vector based on a bidirectional cross-modal attention mechanism; The vector fusion module is used to determine a fusion vector according to a target text vector, a target image vector and a target video vector, and to construct a knowledge graph according to the fusion vector.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it implements the knowledge graph construction method as described in any one of claims 1-7.

10. A storage medium storing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the knowledge graph construction method as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Attention mechanism-based intention recognition method and device, equipment and storage medium

    CN111737458A

  • Multi-modal knowledge graph construction method

    CN112200317A

  • Power grid dispatching multi-mode knowledge graph construction method and system

    CN118035463A

  • Intention identification method, apparatus and device based on attention mechanism, and storage medium

    WO2021232589A1