Data mining methods, query methods and question-answering methods based on knowledge graphs

Abstract

The present application relates to a data mining method based on knowledge graph, a multimodal data query method and a vertical domain knowledge question-answering method. The mining method includes: generating a knowledge graph based on a first artificial intelligence model; inputting the initial node information obtained from the knowledge graph into a second artificial intelligence model, wherein the second artificial intelligence model includes a knowledge graph expansion unit, an abnormal node acquisition unit, a key node acquisition unit and a value mining unit; expanding the expanded node information from the initial node information through the knowledge graph expansion unit and outputting the expanded node information to the abnormal node acquisition unit and the key node acquisition unit respectively; outputting the abnormal nodes and key nodes in the knowledge graph to the value mining unit through the abnormal node acquisition unit and the key node acquisition unit respectively, so that the value mining unit presents the relevant information of the abnormal nodes and key nodes.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and specifically to a data mining method based on knowledge graph, a multimodal data query method, and a vertical domain knowledge question and answer method. Background Art

[0002] Cultural property exchanges, collection companies, museums and other cultural institutions often collect digital cultural products with cultural value. These institutions generate massive amounts of data about these digital cultural products in their work, such as text materials, pictures, audio materials, and video materials, that is, there is a large amount of multimodal data. How to analyze and manage such data and further explore the value of data on this basis has become an urgent need in this field. Traditional data processing methods cannot efficiently extract potential associations in data and conduct intelligent value analysis.

[0003] As a technical means to display the relationship between data in a graphical way, knowledge graph can help users understand data intuitively, but the existing knowledge graph technology has shortcomings in deep analysis and data value mining. Summary of the invention

[0004] The present application provides a method that can generate a cross-modal knowledge graph and perform deep data mining based on the knowledge graph to solve or at least partially solve the technical problems existing in the prior art.

[0005] On the one hand, the present application provides a data mining method based on a knowledge graph, comprising:

[0006] S400, generating a knowledge graph based on the first artificial intelligence model;

[0007] S405, inputting the initial node information obtained from the knowledge graph into a second artificial intelligence model, wherein the second artificial intelligence model includes a knowledge graph expansion unit, an abnormal node acquisition unit, a key node acquisition unit, and a value mining unit;

[0008] S410, expanding the expanded node information from the initial node information through the knowledge graph expansion unit, and outputting the expanded node information to the abnormal node acquisition unit and the key node acquisition unit respectively;

[0009] S415. Output the abnormal nodes and key nodes in the knowledge graph to the value mining unit through the abnormal node acquisition unit and the key node acquisition unit respectively, so that the value mining unit presents relevant information of the abnormal nodes and key nodes.

[0010] In a specific embodiment, the key node acquisition unit outputs the key nodes in the knowledge graph to the value mining unit, including:

[0011] Sort the degree centrality of all nodes and select the node with the highest degree centrality;

[0012] Calculate the betweenness centrality of the node with the highest degree centrality among all adjacent nodes;

[0013] Nodes with betweenness centrality higher than a preset threshold are selected as key nodes.

[0014] In a specific embodiment,

[0015] The second artificial intelligence model also includes a prediction unit,

[0016] The method further comprises:

[0017] S420. The prediction unit uses the digital cultural product related nodes obtained from the knowledge graph expansion unit as explanatory variables and the prices of the digital cultural products as explained variables, and analyzes the coefficients of the related nodes to determine the degree of influence of the related nodes on the future prices of the digital cultural products.

[0018] In a specific embodiment,

[0019] The knowledge graph expansion unit includes a knowledge reasoning unit and a node analysis unit;

[0020] The method of expanding the expanded node information from the initial node information by the knowledge graph expansion unit and outputting the expanded node information to the abnormal node acquisition unit and the key node acquisition unit respectively includes:

[0021] S4100, inputting the node information obtained from the knowledge graph into the knowledge reasoning unit to obtain a reasoning result;

[0022] S4105: Input the inference result into the node analysis unit to extract and analyze node features.

[0023] In one embodiment, the method further comprises:

[0024] S4120, using the output of the node analysis unit as the input of the knowledge reasoning unit again;

[0025] Repeat steps S4100-S4120 until a preset number threshold is reached, and use the latest output of the node analysis unit as the output of the knowledge graph expansion unit.

[0026] In a specific embodiment, causing the value mining unit to present relevant information of the abnormal nodes and key nodes includes:

[0027] Present abnormal node pages and key node pages through value mining units;

[0028] In response to a user clicking on a node in the abnormal node page and the key node page, information related to the node is obtained from the knowledge graph, and related data is searched for in the multimodal data source.

[0029] In a specific embodiment, the step of causing the value mining unit to present the relevant information of the abnormal nodes and the key nodes further includes:

[0030] The value mining unit monitors the user's browsing of the searched related data in real time to determine whether the user is interested;

[0031] If interested, the content of other multimodal data based on the node corresponding to the related data as a common tag is provided to the user.

[0032] In a specific embodiment, generating a knowledge graph based on the first artificial intelligence model includes:

[0033] Obtain vertical domain multimodal data to be classified;

[0034] Inputting the multimodal data into a first artificial intelligence model to generate cross-modal labels corresponding to each of the multimodal data, wherein the first artificial intelligence model is obtained based on the base large model;

[0035] Associating the cross-modal tags to a knowledge graph.

[0036] In one embodiment, obtaining the first artificial intelligence model based on the base large model includes:

[0037] The base large model is transformed to form a first target model, wherein the first target model sets a text encoder, an image encoder, an audio encoder and a video encoder at the original input layer of the base large model to extract features of samples of corresponding types; sets an input projector to project the extracted features into a common feature space; uses the interaction between the self-attention mechanism of the base large model and the feedforward network to achieve feature fusion; sets a classifier after the original output layer of the base large model to generate cross-modal labels; sets a knowledge extractor after the classifier to extract entities and relationships from the labels;

[0038] Pre-training the first target model to obtain a pre-trained first target model;

[0039] Fine-tune the pre-trained first target model to obtain the first artificial intelligence model.

[0040] In one embodiment, pre-training the first target model to obtain a pre-trained first target model includes:

[0041] Acquire a large-scale multimodal data sample, wherein the large-scale multimodal data sample includes a large-scale text sample, a large-scale image sample, a large-scale audio sample, and a large-scale video sample;

[0042] The first target model is pre-trained based on the large-scale multimodal data sample to obtain the first target model.

[0043] In one embodiment, fine-tuning the pre-trained first target model to obtain the first artificial intelligence model includes:

[0044] Acquire vertical domain related multimodal data samples, wherein the multimodal data samples include vertical domain text samples, vertical domain picture samples, vertical domain audio samples and vertical domain video samples;

[0045] The pre-trained first target model is fine-tuned based on the multimodal data samples to obtain the first artificial intelligence model.

[0046] In one embodiment, fine-tuning the pre-trained first target model based on the multimodal data sample to obtain the first artificial intelligence model includes:

[0047] Providing a transfer learning method library, wherein the method library includes multiple transfer learning methods;

[0048] Calculate the quantity ratio of different types of samples in the obtained vertical domain related multimodal data samples;

[0049] According to the type of the vertical domain-related multimodal data samples obtained, an appropriate transfer learning method is called from the method library to fine-tune the pre-trained first target model to obtain the first artificial intelligence model.

[0050] In one embodiment, the method further comprises:

[0051] Define the entities, relationships and attributes of the knowledge graph according to the specification constraints of the vertical domain;

[0052] Associating the cross-modal tag with the knowledge graph includes:

[0053] Entity recognition and relationship extraction are performed by the knowledge extractor to obtain entities and relationships respectively;

[0054] Link the entities and relations to the knowledge graph.

[0055] In one embodiment, linking the entities and relationships to a knowledge graph includes:

[0056] Determine the degree of agreement between the triples of entities and relations output by the knowledge extractor and the existing triples in the current knowledge graph;

[0057] Based on the degree of fit, link to the current knowledge graph.

[0058] In one embodiment, linking to the current knowledge graph based on the degree of fit includes:

[0059] Select the triple consisting of entities and relations with the greatest degree of fit;

[0060] Inputting the triples consisting of the entities and relations with the greatest degree of fit back into the first artificial intelligence model to perform common sense detection that conforms to the vertical domain;

[0061] If the detection result of the first artificial intelligence model is consistent with common sense, the new triplet is retained to be integrated with the current knowledge graph.

[0062] In one embodiment, the method further comprises:

[0063] If it is detected that the meaning of the cross-modal label has different semantics, outputting the semantics corresponding to the multi-semantic label through the first artificial intelligence model;

[0064] A multi-semantic mapping relationship is formed between the multi-semantic tag and the corresponding multiple semantics, and is stored in a multi-semantic mapping database.

[0065] A second aspect of the present application provides a multimodal data query method, comprising:

[0066] Presenting a query entry and a list of entity names of the knowledge graph obtained by the method of the first aspect on the first human-computer interaction interface;

[0067] In response to an entity name input by a first user at the query entry, presenting a local knowledge graph having an edge relationship with the entity name on the human-computer exchange interface;

[0068] In response to the first user clicking on an entity in the local knowledge graph, the corresponding multimodal data is presented on the human-computer interaction interface in the form of thumbnails.

[0069] The third aspect of the present application provides a vertical domain knowledge question answering method, including:

[0070] Presenting a question-and-answer entrance on the second human-computer interaction interface;

[0071] Receiving a question input by a second user in the question-and-answer portal;

[0072] The answer corresponding to the question is output at the question-and-answer entrance at least based on the first artificial intelligence model and the knowledge graph, wherein the first artificial intelligence model and the knowledge graph are obtained according to the method described in the first aspect of the present application.

[0073] In one embodiment, outputting the answer corresponding to the question at the question-and-answer entrance based at least on the first artificial intelligence model and the knowledge graph includes:

[0074] If the user's query intent is recognized through the first artificial intelligence model, and the entity in the question is determined;

[0075] Link the entities identified in the question with entities in the knowledge graph;

[0076] Check whether the entity in the linked knowledge graph has the identifier.

[0077] If the entity in the linked knowledge graph has no identifier, construct a knowledge graph query statement according to the question intent and the linked entity;

[0078] If the entity in the linked knowledge graph has an identifier, construct a knowledge graph query statement and a multi-semantic mapping database query statement according to the question intent and the linked entity;

[0079] Use the constructed knowledge graph query statement and multi-semantic mapping database query statement to retrieve relevant information in the knowledge graph and multi-semantic mapping database respectively;

[0080] Based on the retrieved relevant information, the answer corresponding to the question is output at the question and answer entrance.

[0081] The fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory, wherein the memory stores computer-executable instructions, and the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the classification method described in the first aspect, the query method described in the second aspect, or the question-and-answer method described in the third aspect.

[0082] The fifth aspect of the present application provides a computer-readable storage medium, in which multiple program codes are stored, and the program codes are suitable for being loaded and run by a processor to execute the classification method described in the first aspect, or the query method described in the second aspect, or the question-and-answer method described in the third aspect.

[0083] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0084] Through the knowledge graph generation method of this application, massive multimodal data can be efficiently classified;

[0085] Through the knowledge graph-based data mining method of this application, it is possible to deeply mine the data in the knowledge graph and provide decision-making for users;

[0086] Through the query method of the present application, files can be efficiently queried and managed.

[0087] Through the question-answering method of the present application, knowledge in a certain field can be understood efficiently and comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0089] Figure 1 is a schematic diagram of a hardware architecture for implementing a method of the present application according to an embodiment of the present application;

[0090] Figure 2 is a schematic diagram of a model network topology according to an embodiment of the present application;

[0091] Figure 3 It is a flowchart of the main steps of a data mining method based on a knowledge graph according to an embodiment of the present application. DETAILED DESCRIPTION

[0092] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0093] In the description of this application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Computer-readable storage media include any suitable media that can store program codes, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc.

[0094] The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one of A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B, or A and B. The singular forms "a", "the" and "the" may also include plural forms.

[0095] Here we first explain some terms involved in this application.

[0096] Large language model: refers to a model trained using deep learning technology that can process large-scale data.

[0097] Multimodal large language model: Able to process and understand data of multiple different modalities at the same time. Multimodal large language model can use data of multiple modalities for joint modeling and prediction at the same time, so as to understand and process the input information more comprehensively.

[0098] Vertical Domain: refers to the requirements and characteristics for a specific industry or application field. These requirements are usually industry-specific, such as medical, financial, telecommunications, etc. Software architecture solutions in vertical domains are usually designed to meet the business process and regulatory requirements of a specific industry.

[0099] Base large model: refers to a general large language model that has not undergone additional instruction fine-tuning or downstream task adaptation fine-tuning.

[0100] Vertical domain big model: refers to the "special model" for a specific field obtained by fine-tuning or adapting the base big model using data samples from a specific field.

[0101] Knowledge graph: refers to a technical method that uses a graph model to describe knowledge and model the relationship between all things in the world. It consists of nodes and edges. Among them, nodes represent entities, concepts or attribute values, edges represent relationships, and entities and relationships form triples.

[0102] Multimodal knowledge graph: refers to the presentation of knowledge in a way that expands a single text-based node into multimodal data including digital images, audio, etc. Multimodal data is integrated with the corresponding entities, relationships, and attributes of the text to complete the construction of a multimodal knowledge graph.

[0103] Digital cultural products: products with cultural value that citizens, legal persons and other organizations can create, copy and collect in accordance with the law, such as digitized products of intangible cultural heritage works.

[0104] Figure 1 A structural block diagram of a computer system provided by an exemplary embodiment of the present application is shown, and the method involved in the present application is executed by the computer system or part of the computer system.

[0105] The computer system includes: a first server 12, a first terminal 14, a second server 10, and a second terminal 17. The first terminal 14 is connected to the first server 12 via a wireless network or a wired network. The first server is connected to the remote second server 10 via a wireless network. The second terminal 17 is also connected to the remote second server 10 via a wireless network.

[0106] The first server 12 and the second server 10 may each include at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server includes a processor and a memory.

[0107] The first terminal 14 and the second terminal 17 can be desktop computers or portable mobile terminals, such as smart phones, tablet computers, laptop computers, etc., as long as they have display and input functions. The first terminal 14 and the second terminal 17 also include a processor and a memory.

[0108] Among them, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations such as large model modeling and training.

[0109] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory is used to store at least one instruction, which is used to be executed by the corresponding processor to implement the method involved in the present application.

[0110] In a specific application scenario, it involves a model developer, a model demander, and users of the demander. Among them, the model demander is the cultural field unit such as the cultural property exchange, collectibles company, museum, etc. described in the background technology. They need to analyze and manage the massive data of the unit and further mine the data value on this basis. The various multimodal data of the unit may be distributed in the respective computers of employees 16. The second server 10 is the server of the model developer, which has strong computing power. For example, it has an AI processor, which is used to perform secondary development based on the existing open source base model, build a new network architecture for downstream tasks, and then use vertical domain data samples (such as cultural field data samples) to train the new network architecture to obtain the artificial intelligence model of this application. Finally, the artificial intelligence model of this application is used to manage various multimodal data of the demander unit, mine data value, and even give investment strategy recommendations based on data value mining.

[0111] Optionally, the artificial intelligence model can still be deployed in the second server 10 after training, and various multimodal data of the demand-side unit can be uploaded to the second server 10 through the first server 12 for management, and the model can perform data value mining and provide investment strategy recommendations. The employee 16 can receive model feedback through the relevant software on his first terminal 14, for example, through the account assigned to him.

[0112] Alternatively, after training, the artificial intelligence model can be deployed in the demand-side unit's own server, that is, in the first server 12. The advantage of this is better confidentiality.

[0113] Alternatively, if the computing power of the demand side's own server is good, the work of secondary development, construction and training of a new network architecture for downstream tasks based on the existing open source base model can also be completed on its own server. In this case, the participation of the second server 10 may not be required.

[0114] Alternatively, in order to reduce the computational burden of the first server 12, a distilled artificial intelligence model (student model) can be obtained from the artificial intelligence model (teacher model) trained on the second server 10 by a knowledge distillation method, so that the distilled student model is deployed on the first server 12. In this application, the distilled student model is still referred to as an artificial intelligence model, and generally no distinction is made unless it is necessary to explain the difference between the two.

[0115] On the other hand, after the demand-side unit obtains the model feedback results, the present application also provides an interface for querying the model feedback results to the external user 18 of the demand-side unit, and the external user 18 can obtain it through his own terminal device 17. Alternatively, the demand-side unit can develop a cultural asset display all-in-one machine with a server and display and input functions, such as placing it in an exhibition hall, for query by the external user 18. In this case, the terminal device 17 is not needed.

[0116] Model network topology

[0117] See attached Figure 2 , Figure 2 It is a model network topology structure according to an embodiment of the present application, wherein the artificial intelligence model involved in the present application mainly includes two parts, one is a first artificial intelligence model 2 for generating a knowledge graph, and the other is a second artificial intelligence model 5 for data mining based on the obtained knowledge graph.

[0118] The first artificial intelligence model network topology and training

[0119] For the classification (i.e. labeling) of multimodal data, the existing general large language model is not satisfactory, but the large language model has a strong semantic understanding ability. For this reason, the inventors chose an open source base large model for secondary development, such as using ChatGLM as the base large model. ChatGLM is an open language model based on the general language model GLM framework.

[0120] In order to meet the needs and facilitate the classification query of one's own multimodal data, this application constructs a new network topology 2 based on the selected base model.

[0121] The ChatGLM model is designed based on the Transformer architecture and includes an encoder and a decoder. ChatGLM is mainly used to process natural language tasks. Its core advantage lies in understanding and generating text rather than directly processing multimodal data. Therefore, in order to process multimodal feature extraction, as shown in the figure, an additional encoder module is added to the original input layer 20 of the ChatGLM model, including a text encoder 21, an image encoder 22, an audio encoder 23, and a video encoder 24.

[0122] Among them, the text encoder 21, such as the BERT model, is used to extract the semantic features of text data; the image encoder 22, such as the CNN model, is used to extract the visual features of picture data; the audio encoder 22, such as Wave2Vec, is used to extract the audio features of audio data, such as Mel-frequency cepstral coefficients (MFCC); the video encoder 24, such as a two-stream network (Two-Stream Networks), is used to extract the visual and motion features of video data.

[0123] The extracted features of each modality need to be projected into a common feature space for effective fusion. To this end, an input projector 25 is added to map the features of different modalities into a representation aligned with the text feature space, which can be achieved through a linear layer, MLP.

[0124] Feature fusion is achieved by using the interaction between the ChatGLM model’s own self-attention mechanism and the feedforward network, which is indicated by reference numeral 26 in the figure. The self-attention mechanism allows the model to dynamically assign different weights between different words, thereby achieving feature fusion. The feedforward network further performs nonlinear transformations on these fused features to enhance the model’s expressive power.

[0125] The output layer 27 of the ChatGLM model generates a log probability distribution, which is a vector of raw scores for all possible categories. A classifier 28 is added after the output layer 27, for example, a softmax function is applied to the output layer. The Softmax function converts these raw scores into a probability distribution so that the sum of the probabilities of all categories is 1. In this way, each category has a predicted probability between 0 and 1. Based on the output of the classifier 28, the category with the highest probability is selected as the final classification result. This step is usually implemented by the argmax function, which returns the index of the category with the highest probability.

[0126] At this point, the data of each modality is labeled with semantic labels, which have the nature of "cross-modality". This is because due to the existence of encoders for each modality of data and subsequent feature fusion, the data features of different modalities are comprehensively considered, making the generated cross-modal labels more accurate. In other words, in this process, data features of different modalities such as text, audio, pictures, and videos are integrated into a unified feature space, influencing each other and jointly determining the final cross-modal label.

[0127] For example, when generating cross-modal tags for text, the constructed model will comprehensively consider the features of the text and the features of audio, pictures, and videos. This allows the cross-modal tags of text to contain not only the information of the text itself, but also the relevant information of multi-modal data such as audio, pictures, and videos. Similarly, the generation of cross-modal tags for audio and video will also be affected by the features of other modal data.

[0128] The granularity and form of the labels for each modal data may be different. For example, for the text modality, the cross-modal labels generated are usually regarded as one or more words, because the main function of text is to express meaning, and usually only a concise and clear word or phrase is needed to convey the information. For audio or video modalities, the generated cross-modal labels may be more inclined to the phrase or sentence level, because the data of these modalities usually contain a large number of audio frames or video frames, so longer sequences can be extracted from them to generate corresponding cross-modal labels. Of course, this is not absolute. In some scenarios, such as when complex audio or video content needs to be classified, longer or more complex label sequences may be generated. At the same time, for various modal data, the granularity and form of the labels can also be adjusted according to actual needs and application scenarios.

[0129] In order to facilitate inquiries by employees of the demand-side unit, this application hopes to associate the cross-modal tags of each multimodal data with the knowledge graph 3.

[0130] In the ChatGLM network architecture, the ChatGLM model is a unidirectional language model that is mainly used to predict the next word, rather than to perform entity recognition and relation extraction for a given tag. Therefore, it is necessary to combine other technologies and methods to achieve this. Figure 2 The knowledge extractor 29 was introduced into the first artificial intelligence model.

[0131] In one example, LSTM (Long Short-Term Memory) is used as a knowledge extractor. LSTM is a common sequence model that can be used to process sequence data. Since what needs to be processed is a label (a word or sentence, which is essentially a text sequence), LSTM is suitable. The LSTM model can be used to model entity recognition and relationship extraction problems in text sequences. Specifically, the text sequence can be regarded as a sequence data, and LSTM can be used to capture the long-term dependencies in the sequence.

[0132] In the ChatGLM network architecture, we do not need to use LSTM at every position, but can only use it at the specific position of interest. In other words, an additional LSTM module can be added outside the output layer of the ChatGLM model to model specific entity recognition and relationship extraction tasks. This module can share part of the ChatGLM model, thereby reducing the number of model parameters and computational complexity.

[0133] Next, the newly constructed network architecture needs to be trained to ultimately obtain the first artificial intelligence model.

[0134] The training of the newly constructed network architecture (the first artificial intelligence network before training is called the first target model) generally includes two processes: S10 pre-training and S20 fine-tuning training.

[0135] Specifically, the steps of S10 pre-training include:

[0136] S100. Acquire a large-scale multimodal data sample, wherein the large-scale multimodal data sample includes a large-scale text sample, a large-scale picture sample, a large-scale audio sample, and a large-scale video sample.

[0137] These large-scale samples are manually annotated samples and can be obtained through the Internet or commercial databases.

[0138] S105. Pre-train the first target model based on the large-scale multimodal data sample to obtain a pre-trained first target model.

[0139] The original multimodal data samples need to be converted into a representation suitable for processing in ChatGLM. For data in different modalities such as text, audio, and video, corresponding preprocessing techniques can be used for processing.

[0140] The loss function of the training uses a cross entropy loss function. In one example, the training stops when the loss is less than a preset threshold.

[0141] The steps of S20 fine-tuning training include:

[0142] S200: Acquire vertical domain related multimodal data samples, wherein the multimodal data samples include vertical domain text samples, vertical domain picture samples, vertical domain audio samples, and vertical domain video samples.

[0143] In one embodiment of the present application, the vertical domain is the field of cultural industry.

[0144] The vertical domain-related multimodal data samples may be provided by demand-side customers in the relevant fields.

[0145] S205. Fine-tune the pre-trained first target model based on the multimodal data sample to obtain the first artificial intelligence model, i.e., the vertical domain large model.

[0146] In one embodiment, step S205 further includes:

[0147] S2050. Provide a transfer learning method library, wherein the method library includes multiple transfer learning methods.

[0148] For example, the strategy library includes P-Tuning method, LoRA method, full parameter training method, Prompt Tuning method, Feature Adaption method, etc.

[0149] S2052. Calculate the quantity ratio of different types of samples in the obtained vertical domain related multimodal data samples.

[0150] S2054. According to the type of the acquired vertical domain-related multimodal data samples, call an appropriate transfer learning method from the method library to fine-tune the pre-trained first target model to obtain the first artificial intelligence model.

[0151] For example, among the vertical domain related multimodal data samples obtained, image samples are the most numerous. In this case, the full parameter training method is selected to fine-tune the image encoder part to make full use of a large amount of data to improve the performance of the model in image recognition tasks.

[0152] For example, if the number of text samples is moderate, the Prompt Tuning method is selected to optimize the prompt to adapt to the specific text understanding task. This method is parameter-efficient and suitable for medium-sized data.

[0153] For example, if the number of audio texts is small, the LoRA method or the P-Tuning method can be selected. These methods have advantages in parameter efficiency, are suitable for fine-tuning of small-scale data, and can reduce the risk of overfitting.

[0154] Generally speaking, in the prior art of fine-tuning the model using the transfer learning method, a fixed transfer learning method is used. However, this application fully pays attention to the fact that multimodal samples are provided by various demand-side units, but in practice, it is found that the proportion of multimodal samples that can be provided by various demand-side units is not the same, some have more video samples, and some have more text samples. In this case, if a transfer learning method is used, the obtained vertical model will not be targeted, affecting the customer experience.

[0155] Additionally, in one embodiment, Figure 2A reward model can also be deployed in the first artificial intelligence model for training in an RLHF reinforcement learning manner to fine-tune some parameters of the first target model mentioned above. RLHF allows humans to directly guide machines and allows machines to master decision-making elements that are clearly embedded in human experience. As an effective alignment technology, RLHF can mitigate the harmful content generated by large models to a certain extent and improve information integrity.

[0156] Multi-semantic mapping relationship

[0157] In the process of training the above-mentioned first target model using samples from the cultural field, the inventors found that the meanings of some labels have different semantics in different contexts. For example, the semantics of the generated label "Spring and Autumn" include historical periods, seasons, and even book titles. This may cause inaccurate returned results when performing retrieval queries in the future. For this reason, the ChatGLM model of the present application can also output the semantics corresponding to labels with multiple semantics. Furthermore, the present application forms a multi-semantic mapping relationship between the multi-semantic labels and the corresponding multiple semantics, and stores them in the multi-semantic mapping database 4, such as Figure 2 At the same time, the knowledge graph entity corresponding to the multi-semantic label is marked. The role of the multi-semantic mapping database 4 will be further introduced later when the external user 18 asks questions.

[0158] Defining the knowledge graph structure

[0159] To design a knowledge graph, it is necessary to define entities and relationships based on the requirements of the demander (often subject to the norms of the field, such as common classification patterns, etc.).

[0160] Entity definition: can be used as a node type in the knowledge graph.

[0161] Relationship definition: defines the relationship between entities.

[0162] Taking the demand side unit as the Cultural Property Exchange as an example, we design a knowledge graph for its digital cultural products:

[0163] 1. Design entity

[0164] Cultural relics: including all kinds of movable cultural relics, such as porcelain, calligraphy and painting, sculpture, etc. Attributes include name, age, type, size, weight, etc.

[0165] Production time: the year when the cultural relic was produced, such as "Tang Dynasty", "Song Dynasty", etc.

[0166] Production technology: the production technology of cultural relics, such as "blue and white porcelain", "cloisonné", etc.

[0167] Pattern features: pattern styles on cultural relics, such as "lotus pattern", "dragon pattern", etc.

[0168] Production place: the place where the cultural relic was produced, such as "Jingdezhen", "Suzhou", etc.

[0169] Source of raw materials: the origin of the raw materials used to make cultural relics, such as "kaolin", "Hetian jade", etc.

[0170] Technological evolution: the evolution of artifact production technology over time.

[0171] Cultural influence: the influence of cultural relics on later cultures, such as "influence on European porcelain production".

[0172] 2. Design relationships

[0173] Made of: indicates that the cultural relic is made of specific raw materials, such as "blue and white porcelain is made of kaolin."

[0174] Made in...period: indicates the period when the cultural relic was made, such as "blue and white porcelain was made in the Ming Dynasty."

[0175] Using...technology: refers to the production technology used in the cultural relics, such as "blue and white porcelain uses blue and white technology".

[0176] Having...characteristics: refers to the pattern characteristics of cultural relics, such as "blue and white porcelain has lotus pattern characteristics."

[0177] Produced in...Place: Indicates the place where the cultural relic was produced, such as "blue and white porcelain was made in Jingdezhen".

[0178] Technology evolved from: Indicates the relationship of technological evolution, such as "Cloisonné craft technology evolved from blue and white porcelain craft."

[0179] Influenced...culture: indicates the influence of cultural relics on culture, such as "blue and white porcelain influenced European porcelain culture."

[0180] Second artificial intelligence model network topology and training

[0181] like Figure 2 As shown, the second artificial intelligence model 5 includes: a knowledge reasoning unit 51, a node analysis unit 52, an abnormal node acquisition unit 53, a key node acquisition unit 54 and a value mining unit 55.

[0182] Among them, the knowledge reasoning unit 51 discovers new knowledge that is not displayed in the knowledge graph based on the node information obtained in the knowledge graph 3.

[0183] In a specific embodiment, the methods used by the knowledge reasoning unit 51 include logic-based reasoning, graph-based reasoning, machine learning-based reasoning, etc. For example, logic-based methods, such as first-order predicate logic, description logic, etc.; graph-based methods, such as Path Ranking Algorithm (PRA) and Association Rule Mining under Incomplete Evidence (AMIE); machine learning-based reasoning, for example, is implemented using models such as TransE and TransH.

[0184] In another specific embodiment, the strategies of the above multiple reasoning methods can also be combined to form a hybrid reasoning mechanism to fully utilize the advantages of various methods and make up for the shortcomings of each other. The hybrid reasoning mechanism can achieve a more comprehensive and efficient effect in processing the cross-modal knowledge graph reasoning tasks involved in this application.

[0185] The node analysis unit 52 is connected to the knowledge reasoning unit 51 and takes the reasoning result of the knowledge reasoning unit 51 as input to extract and analyze node features.

[0186] In a specific embodiment, the node analysis unit 52 can be implemented by a graph neural network (GNN). Specifically, the inference result obtained by the knowledge inference unit 51 is input into the GNN model, wherein each of the two expanded nodes (i.e., entities, including nodes of the original knowledge graph and new nodes added by reasoning) obtained by reasoning is regarded as a pixel, and each of the two expanded relationships (i.e., edges, including relationships of the original knowledge graph and new relationships added by reasoning) obtained by reasoning is regarded as a channel and input into the GNN model, so as to analyze the attributes of the node itself and its adjacency relationship.

[0187] In a preferred embodiment, the output of the GNN can be used as the input of the knowledge reasoning unit 51 again, that is, the output of the node analysis unit is used for knowledge reasoning, forming a closed-loop collaborative workflow. Knowledge reasoning focuses on logic and rules, while GNN focuses on learning patterns and features from data. Through the joint use of the two, knowledge reasoning discovers new knowledge, and GNN can extract and utilize this knowledge for deeper analysis, and then provide the analysis results to the knowledge reasoning unit to prompt the knowledge reasoning unit to infer more new knowledge (form new nodes and edges).

[0188] The knowledge reasoning unit 51 and the node analysis unit 52 can be collectively referred to as a knowledge graph expansion unit 50. By setting up the knowledge reasoning unit 51 and the node analysis unit 52, it is possible to further mine hidden information that is not reflected in the knowledge graph originally obtained by the first artificial intelligence model, thereby providing solid technical support for comprehensive abnormal node analysis and key node analysis, and on this basis, the effect of data value mining and prediction can be guaranteed.

[0189] The abnormal node acquisition unit 53 and the key node analysis unit 54 respectively receive the output of the knowledge graph expansion unit 50. In the embodiment where the knowledge reasoning unit 51 and the node analysis unit 52 interact, a threshold of the number of interactions between the two can be set. Once the threshold is met, the latest output of the node analysis unit 52 is finally used as the output of the knowledge graph expansion unit 50.

[0190] The abnormal node acquisition unit 53 can use clustering algorithms or abnormality detection technology to mark "abnormal" nodes in the knowledge graph.

[0191] For clustering algorithms, taking the clustering of cultural relic pattern features as an example, the feature vectors of each cultural relic can be quantized, and then all feature vectors can be divided into different clusters. Next, the similarity between the cultural relics in each cluster is calculated, and then the group of cultural relics with the lowest similarity is selected as anomalies. Specifically, K-means clustering algorithm, spectral clustering or other clustering algorithms can be used.

[0192] In K-means clustering, the feature vector of each artifact can be regarded as a point and then divided into the nearest K clusters; in spectral clustering, the feature vector of the artifact can be mapped to a high-dimensional feature space, and then the nearest neighbors can be found in the space to group similar artifacts into one cluster.

[0193] As for anomaly detection technology, still taking the anomaly detection of pattern features of cultural relics as an example, some well-known anomaly detection algorithms can be used, such as One-Class SVM (One-Class Support Vector Machine), Isolation Forest, etc.

[0194] The key node acquisition unit 54 may use centrality indicators, such as degree centrality, betweenness centrality, and eigenvector centrality, to identify key nodes.

[0195] In the knowledge graph, the higher the degree centrality of each node, the greater its influence in the network, and therefore the higher its accessibility and importance.

[0196] In the knowledge graph, nodes with high betweenness centrality are more likely to become the center of the network, and therefore have a certain importance in some cases.

[0197] In a preferred embodiment, first, a preliminary screening is performed based on degree centrality. The degree centrality of all nodes can be sorted, and then the node with the highest degree centrality can be selected. Then, further screening can be performed based on betweenness centrality. The node with the highest degree centrality can be selected first, and then its betweenness centrality among all adjacent nodes can be calculated. A threshold can be selected, for example, if a node's betweenness centrality is higher than this threshold, we consider it to be a "key" node.

[0198] After the abnormal nodes and key nodes are obtained, they can be presented through the value mining unit 55, which will be explained in the subsequent application stage.

[0199] In some applications, the demand side unit hopes to obtain the factors affecting the price of collectibles (cultural relics) so as to provide investment advice to itself or its external users. In this case, the second artificial intelligence model also includes a prediction unit 56.

[0200] To this end, the output of the knowledge graph expansion unit 50 is also transmitted to the prediction unit 56. The prediction unit 56 can construct a multivariate time series model (such as a vector autoregression (VAR) model), using the cultural relic related nodes (such as production time, production process, etc.) obtained from the knowledge graph expansion unit 50 as explanatory variables and the price of the cultural relic as the explained variable, analyze the coefficients of these factors, and thus determine their degree of influence, so as to make trend analysis and predictions, and provide investment advice.

[0201] The price of cultural relics can be predicted by using the autoregressive integrated moving average model (ARIMA model) or other forecasting models to predict the future price of cultural relics. Then these forecast results can be included in the multivariate time series model as explained variables. In this case, the model will analyze the influence of explanatory variables such as production time and production process on the predicted price. The ARIMA model is constructed based on the historical sales data of the cultural relics. Specifically, the ARIMA model is used to fit and predict the historical sales data to obtain the demand trend and price changes in the future.

[0202] Next, the newly constructed network architecture (the second artificial intelligence network before training is called the second target model) needs to be trained to finally obtain the second artificial intelligence model 5.

[0203] The second target model can adopt conventional back-propagation methods, stochastic gradient descent methods and other methods, which are not limited in this application.

[0204] In addition, since the second target model is deep, the Dropout method can also be used in training to reduce overfitting.

[0205] The training set can be obtained from the data of the industry. The historical sales data required for the ARIMA model can be obtained from the historical transactions of the industry, which can come from museum collections, private collectors, sales records of antique markets, auction records, etc. The historical transaction database 6 is shown in the figure, which contains the latest historical transaction data.

[0206] Application Phase

[0207] Application Phase 1 - Knowledge Graph Generation

[0208] In one embodiment, the knowledge graph generation method S30 of the present application includes:

[0209] S300: Obtain vertical domain multimodal data to be classified.

[0210] The data to be classified is what the demander needs to label, including at least two of text data, image data, audio data and video data.

[0211] In one embodiment, for example, an employee has put multiple types of files with certain relevance in a folder, and uploads data from the folder.

[0212] In one example, the acquired vertical domain multimodal data includes:

[0213] Textual material: Text describing blue and white porcelain. ”

[0214] Picture information: High-definition pictures of blue and white porcelain.

[0215] Audio material: Blue and white porcelain is now collected in a certain trading center. ”

[0216] Video material: A documentary introducing blue and white porcelain.

[0217] S305: Input the multimodal data into a first artificial intelligence model to generate cross-modal labels corresponding to each of the multimodal data.

[0218] Continuing with the previous example, we get the following cross-modal tags:

[0219] Text tags: "Yuan Dynasty", "Blue and White Porcelain", "Jingdezhen";

[0220] Image tags: “Blue and white porcelain;

[0221] Audio tags: "blue and white porcelain", "a trading center";

[0222] Video tags: "Blue and White Porcelain", "Documentary".

[0223] S310: Associating the cross-modal tag with the knowledge graph.

[0224] In one embodiment, step S310 specifically includes:

[0225] S3100, performing entity recognition and relationship extraction through the knowledge extractor to obtain entities and relationships respectively.

[0226] Still continuing with the above example,

[0227] Entity Recognition:

[0228] Node A: Blue and White Porcelain

[0229] Node B: Ming Dynasty

[0230] Node C: Blue and White Craftsmanship

[0231] Node D: Jingdezhen

[0232] Node E: Kaolin

[0233] Node F: Lotus pattern

[0234] Relation Extraction

[0235] Side 1: A-made during the period -B

[0236] Side 2: A-Using... process-C

[0237] Side 3: A-Made in... Location-D

[0238] Side 4: A - Made of - E

[0239] Side 5: A- has... feature - F.

[0240] S3105. Link the entities and relationships to the knowledge graph.

[0241] S31050, determining the degree of agreement between the triples consisting of entities and relations output by the knowledge extractor and the triples already in the current knowledge graph;

[0242] The degree of agreement is determined by cosine similarity.

[0243] S31055. Link to the current knowledge graph based on the degree of fit.

[0244] In one embodiment, if the degree of fit exceeds a preset threshold, it is considered to be a fit, and the new data can be integrated with the data in the existing knowledge graph to enrich and expand the content of the knowledge graph. This helps to achieve information fusion and unified management between different data sources.

[0245] In another embodiment, in order to more accurately link to the current knowledge graph, instead of comparing with a fixed threshold, the following method is used:

[0246] Select the triple consisting of entities and relations with the greatest degree of fit;

[0247] Inputting the triples consisting of the entities and relations with the greatest degree of fit back into the first artificial intelligence model to perform common sense detection that conforms to the vertical domain;

[0248] If the detection result of the first artificial intelligence model is consistent with common sense, the new triple is retained to be integrated with the current knowledge graph. For example, it is common sense that blue and white porcelain is produced in Jingdezhen. After the training of large-scale data samples and vertical domain samples in the early stage, the first artificial intelligence model can better understand the common sense of the vertical domain, so as to better enrich and expand the content of the knowledge graph.

[0249] Continuing with the above example, the following shows how the above example can be integrated into the knowledge graph.

[0250] A (blue and white porcelain)--made in the Ming Dynasty-->B (Ming Dynasty)

[0251] A (blue and white porcelain) - using blue and white craft --> C (blue and white craft)

[0252] A (blue and white porcelain)--made in Jingdezhen-->D (Jingdezhen)

[0253] A (blue and white porcelain) -- made of kaolin --> E (kaolin)

[0254] A (blue and white porcelain) - with lotus pattern characteristics --> F (lotus pattern)

[0255] It should be noted that in the above examples, the cross-modal tags and entities are consistent, and it seems that no entity recognition step is required and they can be used directly as entities. However, as mentioned earlier, for videos, cross-modal tags may be more inclined to the phrase or sentence level, so entity recognition and relationship extraction are required to divide them into independent subjects and objects.

[0256] The extracted entities and relationships between entities are stored in a graph database, such as Neo4j or OrientDB.

[0257] If the entities and relations extracted by the knowledge extractor do not always match the current knowledge graph, for example, one can consider improving the knowledge extractor to improve the accuracy and completeness of the knowledge graph.

[0258] Those skilled in the art will understand that the above-mentioned association action will be performed each time data to be classified is uploaded so that the knowledge graph is continuously updated.

[0259] Application Phase 2 - Application of Knowledge Graph

[0260] In the application stage, Figure 3 As shown, the present application provides a data mining method S40 based on a knowledge graph, comprising:

[0261] S400. Generate a knowledge graph based on the first artificial intelligence model.

[0262] The process of generating the knowledge graph is shown in the above step S30 and its details.

[0263] S405. Input the initial node information obtained from the knowledge graph into the second artificial intelligence model, wherein the second artificial intelligence model includes a knowledge graph expansion unit, an abnormal node acquisition unit, a key node acquisition unit, and a value mining unit.

[0264] S410, expanding the expanded node information from the initial node information through the knowledge graph expansion unit and outputting the expanded node information to the abnormal node acquisition unit and the key node acquisition unit respectively.

[0265] As mentioned above, the expanded node information here includes the node and edge information of the original knowledge graph, as well as the newly expanded node and edge information.

[0266] In one embodiment, the knowledge graph expansion unit 50 includes a knowledge reasoning unit 51 and a node analysis unit 52.

[0267] Therefore, step S410 further includes:

[0268] S4100. Input the node information (nodes and edges) obtained from the knowledge graph 3 into the knowledge reasoning unit 51 to obtain the reasoning result, that is, discover new knowledge that is not displayed in the knowledge graph.

[0269] The specific methods used are described in the topology and training section and will not be repeated here.

[0270] For example, through these reasoning techniques, we can use the relationship between entities to perform knowledge reasoning. For example, based on the relationship between "blue and white porcelain" and "Yuan Dynasty", we can infer that "the Yuan Dynasty was the main development stage of blue and white porcelain."

[0271] S4105. The node analysis unit 52 uses the reasoning result of the knowledge reasoning unit 51 as input to extract and analyze node features.

[0272] The specific methods used are described in the topology and training section and will not be repeated here.

[0273] In a preferred embodiment, step S320 further includes:

[0274] S4120, the output of the node analysis unit 52 is used as the input of the knowledge reasoning unit 51 again, that is, the output of the node analysis unit is used for knowledge reasoning to form a closed-loop collaborative workflow. Repeat steps S4100-S4120 until the preset number of times is reached, and the latest output of the node analysis unit is used as the output of the knowledge graph expansion unit 50.

[0275] Knowledge reasoning focuses on logic and rules, while the node analysis unit focuses on learning patterns and features from data. Through the joint use of the two, knowledge reasoning discovers new knowledge, and the node analysis unit can extract and utilize this knowledge for deeper analysis, and then provide the analysis results to the knowledge reasoning unit to prompt the knowledge reasoning unit to infer more new knowledge (form new nodes and edges).

[0276] By setting up the knowledge reasoning unit 51 and the node analysis unit 52, it is possible to further explore hidden information that is not reflected in the knowledge graph originally obtained through the first artificial intelligence model, thereby providing solid technical support for comprehensive abnormal node analysis and key node analysis, and on this basis, the effect of data value mining and prediction can be guaranteed.

[0277] S415. Outputting the abnormal nodes and key nodes in the knowledge graph to the value mining unit through the abnormal node acquisition unit and the key node acquisition unit respectively, so that the value mining unit presents relevant information of the abnormal nodes and key nodes.

[0278] The specific method used by the abnormal node acquisition unit 53 is described in the topology and training section and will not be repeated here.

[0279] The specific method used by the key node acquisition unit 54 is described in the topology and training section and will not be repeated here.

[0280] In a preferred embodiment, the acquisition of key nodes includes:

[0281] First, sort the degree centrality of all nodes and select the node with the highest degree centrality;

[0282] Next, the betweenness centrality of the node with the highest degree centrality among all adjacent nodes is calculated;

[0283] Finally, nodes with betweenness centrality above a preset threshold are selected as “key” nodes.

[0284] The combination of the two methods can obtain key nodes more accurately.

[0285] When the second artificial intelligence model further includes a prediction unit, the method further includes:

[0286] S420. The prediction unit uses the digital cultural product related nodes obtained from the knowledge graph expansion unit as explanatory variables and the prices of the digital cultural products as explained variables, and analyzes the coefficients of the related nodes to determine the degree of influence of the related nodes on the future prices of the digital cultural products, thereby making trend analysis and predictions.

[0287] The output of the knowledge graph expansion unit 50 is also transmitted to the prediction unit 56, which uses the nodes obtained from the knowledge graph expansion unit 50 that have an edge relationship with the node name of the digital cultural product as explanatory variables (for example, to predict what factors will affect the future price of blue and white porcelain, relevant nodes with edge connections with the node of blue and white porcelain are obtained from the knowledge graph expansion unit 50), and the price of the digital cultural product as the explained variable, and analyzes the coefficients of these nodes to determine the degree of influence of the nodes on the future price of the digital cultural product. Once you know what factors are at work, it is easy to make trend analysis and predictions, and provide investment advice.

[0288] The future price change trend of digital cultural products is predicted by a prediction model based on data obtained in real time from the historical sales database 6 .

[0289] In one embodiment, causing the value mining unit to present relevant information of the abnormal nodes and key nodes includes:

[0290] S500 , presenting an abnormal node page and a key node page through the value mining unit 55 .

[0291] In a specific embodiment, the value mining unit 55 presents an abnormal node page, which contains all identified abnormal nodes, and on this page, more detailed information about the node can be displayed, such as the production process, historical status, cultural value, etc. At the same time, a timeline or relationship network can be drawn on the page, so that users can better understand and explore the particularity of the node.

[0292] Similarly, for key nodes, the key node page can also be presented through the value mining unit 55.

[0293] In this way, users can comprehensively understand the important nodes and abnormal nodes in the knowledge graph of blue and white porcelain through different views and pages, so as to better understand and explore the historical and cultural value of blue and white porcelain.

[0294] S505. In response to a user clicking on a node in the abnormal node page and the key node page, information related to the node is obtained from the knowledge graph, and related data is searched for in the multimodal data source.

[0295] When a user clicks on a node on a key node page or an abnormal node page, a deep intelligent analysis is triggered. For example, the value mining unit 55 obtains information related to the node from the knowledge graph, such as name, description, location, time, etc. Then, with this information, the system can search for relevant documents or videos in the multimodal data source. If there are multiple matches, the system will also consider other factors, such as popularity, ratings, views, etc., to determine the final recommendation list. The value mining unit 55 can display the recommendation results to the user in the form of a card. The user can view the title, introduction, text, rating and other information of each article or video, and choose whether to read it according to their interests and needs. According to the label of the node, recommend related articles, videos and other content, or directly start a detailed study of the node.

[0296] S510, monitoring the browsing of the searched related data by the user in real time through the value mining unit 55, and determining whether the user is interested;

[0297] S515: If interested, provide the user with content of other multimodal data based on the nodes corresponding to the relevant data as common tags.

[0298] In one embodiment, the value mining unit 55 can monitor in real time the areas that the user may be interested in. For example, if the multimodal data browsed by the user is a video data of blue and white porcelain, and it is detected that the progress bar of the video is dragged back or paused, it is determined that the user is interested in the video, and other multimodal data with the node "blue and white porcelain" as a label are searched. For example, if there is a text data with the label "blue and white porcelain", the text is provided to the user to provide more background knowledge and analysis, so that the user can have a more comprehensive understanding of the content that he / she is interested in.

[0299] It should be noted that the "user" mentioned in the above embodiment can be either an internal employee 16 of the demand-side unit or an external user 18 of the demand-side unit. In other words, the present application provides both types of people with the function of obtaining value mining and the function of prediction.

[0300] In addition to abnormal nodes and key nodes in the knowledge graph, in the present application, the value mining unit 55 also provides demand-side staff with the function of efficiently querying all nodes in the knowledge graph and managing files.

[0301] To this end, a query software system is developed, and the employee 16 can query by logging into his own account. Specifically, an embodiment of the present application provides a multimodal data query method, including:

[0302] S600, presenting a query entry and a list of entity names of the knowledge graph on a human-computer interaction interface;

[0303] S605: In response to the entity name input by the employee at the query entry, the value mining unit presents a local knowledge graph having an edge relationship with the entity name on the human-computer exchange interface;

[0304] S610: In response to an employee clicking on an entity in the local knowledge graph, presenting corresponding multimodal data on the human-computer interaction interface in the form of thumbnails.

[0305] The user can further click on the thumbnail to generate a preview. Further, if the user confirms interest, they can also download it if their personal permissions allow.

[0306] Furthermore, in order to allow external users to better understand the knowledge in the vertical domain, a vertical domain (such as cultural field) knowledge question and answer system has been developed. External users 18 can log in to the system through the second terminal 17 or the aforementioned integrated machine to ask questions.

[0307] Specifically, an embodiment of the present application provides a vertical domain knowledge question answering method, including:

[0308] S700, presenting a question-and-answer entrance on the human-computer interaction interface;

[0309] S705, receiving a question input by an external user in the question-and-answer portal;

[0310] S710. Output the answer corresponding to the question at the question-and-answer entrance based at least on the first artificial intelligence model and the knowledge graph.

[0311] In one embodiment, S710 further includes:

[0312] S7100: If the query intention of the user is identified through the first artificial intelligence model, and the entity in the question is determined;

[0313] S7105, linking the entities identified in the question with the entities in the knowledge graph;

[0314] S7110. Detect whether the entity in the linked knowledge graph has the identifier.

[0315] S7115. If the entity in the linked knowledge graph has no identifier, construct a knowledge graph query statement according to the question intent and the linked entity;

[0316] S7120: If the entity in the linked knowledge graph has an identifier, construct a knowledge graph query statement and a multi-semantic mapping database query statement according to the question intent and the linked entity;

[0317] S7125. Use the constructed knowledge graph query statement and multi-semantic mapping database query statement to retrieve relevant information in the knowledge graph and multi-semantic mapping database respectively;

[0318] S7130. Output the answer corresponding to the question at the question and answer entry according to the retrieved relevant information.

[0319] Queries in the knowledge graph can be implemented through operations such as graph traversal and pattern matching;

[0320] Queries in the multi-semantic mapping database can be implemented through operations such as vector Euclidean similarity.

[0321] In this embodiment, it is important to distinguish whether the keyword in the question has multiple semantics in the entity in the knowledge graph. If there are multiple semantics, there will be misjudgment by simply searching the knowledge graph. For example, the question asked by the user is "I want to know what clothes to wear in Spring and Autumn". In this sentence, Spring and Autumn may have two semantics, one is the historical period, and the other is the season. At this time, the mapping relationship between the entity and the semantics must be called for judgment.

[0322] Of course, those skilled in the art will understand that although the vertical domain knowledge question answering method is described in a manner prepared for external users 18, it does not prevent internal employees 16 from using the method, and they only need to be assigned an external user identity.

[0323] It is understood by those skilled in the art that all or part of the processes in the various methods of the application stage of the above-mentioned embodiments can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0324] The present application also provides an electronic device. Figure 1 The server 12 shown can be used to execute the above classification method; for example Figure 1 The first terminal 14 shown can be used to execute the above query method; for example Figure 1 The second terminal 17 in the embodiment can be used to execute the above question-answering method.

[0325] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

Claims

1. A data mining method based on knowledge graph, characterized in that: include: S400, generating a knowledge graph based on the first artificial intelligence model; S405, inputting the initial node information obtained from the knowledge graph into a second artificial intelligence model, wherein the second artificial intelligence model includes a knowledge graph expansion unit, an abnormal node acquisition unit, a key node acquisition unit, and a value mining unit; S410, expanding the expanded node information from the initial node information through the knowledge graph expansion unit, and outputting the expanded node information to the abnormal node acquisition unit and the key node acquisition unit respectively; S415, outputting the abnormal nodes and the key nodes in the knowledge graph to the value mining unit through the abnormal node acquisition unit and the key node acquisition unit, respectively, so that the value mining unit presents the relevant information of the abnormal nodes and the key nodes; The knowledge graph expansion unit includes a knowledge reasoning unit and a node analysis unit; The method of expanding the expanded node information from the initial node information by the knowledge graph expansion unit, and outputting the expanded node information to the abnormal node acquisition unit and the key node acquisition unit respectively, includes: S4100, inputting the node information obtained from the knowledge graph into the knowledge reasoning unit to obtain a reasoning result; S4105, inputting the inference result into the node analysis unit to extract and analyze node features; S4120, using the output of the node analysis unit as the input of the knowledge reasoning unit again; Repeat steps S4100-S4120 until a preset number of times is reached, and use the latest output of the node analysis unit as the output of the knowledge graph expansion unit; The step of causing the value mining unit to present relevant information of the abnormal nodes and key nodes includes: Present abnormal node pages and key node pages through value mining units; In response to a user clicking on a node in the abnormal node page and the key node page, obtaining information related to the node from the knowledge graph, and searching for related data in the multimodal data source; The step of causing the value mining unit to present the relevant information of the abnormal nodes and key nodes further includes: The value mining unit monitors the user's browsing of the searched related data in real time to determine whether the user is interested; If interested, the content of other multimodal data based on the nodes corresponding to the related data as common tags is provided to the user.

2. The method according to claim 1, characterized in that The second artificial intelligence model further includes a prediction unit, and the method further includes: S420. The prediction unit uses the digital cultural product related nodes obtained from the knowledge graph expansion unit as explanatory variables and the prices of the digital cultural products as explained variables, and analyzes the coefficients of the related nodes to determine the degree of influence of the related nodes on the future prices of the digital cultural products.

3. The method according to any one of claims 1 to 2, characterized in that: The generating of a knowledge graph based on the first artificial intelligence model includes: Obtain vertical domain multimodal data to be classified; Inputting the multimodal data into a first artificial intelligence model to generate cross-modal labels corresponding to each of the multimodal data, wherein the first artificial intelligence model is obtained based on the base large model; Associating the cross-modal tags to a knowledge graph.

4. The method according to claim 3, characterized in that The method further comprises: Define the entities, relationships, and attributes of the knowledge graph according to the normative constraints of the vertical domain; Associating the cross-modal tag with the knowledge graph includes: Entity recognition and relationship extraction are performed through the knowledge extractor to obtain entities and relationships respectively; Link entities and relations to the knowledge graph.

5. A multimodal data query method, characterized in that: include: Presenting a query entry and a list of entity names of the knowledge graph obtained by the method according to any one of claims 1 to 4 on a first human-computer interaction interface; In response to an entity name input by a first user at the query entry, the value mining unit presents a local knowledge graph having an edge relationship with the entity name on the human-computer interaction interface; In response to the first user clicking on an entity in the local knowledge graph, the corresponding multimodal data is presented on the human-computer interaction interface in the form of thumbnails.

6. A vertical domain knowledge question answering method, characterized in that: include: Presenting a question-and-answer entrance on the second human-computer interaction interface; Receiving a question input by a second user in the question-and-answer portal; The answer corresponding to the question is output at the question-and-answer entrance based at least on the first artificial intelligence model and the knowledge graph, wherein the first artificial intelligence model and the knowledge graph are obtained according to the method described in any one of claims 1-4.

Images