Multi-modal data automatic classification method, electronic equipment and storage medium

Abstract

The invention relates to a multi-modal data automatic classification method, electronic equipment and a storage medium, and particularly provides a multi-modal data automatic classification method, which comprises the following steps: acquiring vertical domain multi-modal data to be classified; inputting the multi-modal data into a vertical domain model, generating cross-modal labels corresponding to the multi-modal data, and obtaining the vertical domain model based on a base large model; and associating the cross-modal label to a knowledge graph.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a multimodal data automatic classification method, electronic device and storage medium. Background Art

[0002] In the daily work of cultural enterprises and institutions such as museums, the document data that needs to be managed is often of many types and quantities, such as text files, image files, video files, audio files, etc. The massive multimodal data cannot be effectively classified and managed.

[0003] In the prior art, one way is to have employees manually review and analyze each file one by one, and manually name the files according to the content. This method is usually time-consuming and labor-intensive and is easily affected by human factors, resulting in inconsistent or inaccurate naming. Another method is a keyword matching algorithm, which automatically matches and tags the keywords in the file through a predefined keyword library. Although this method improves efficiency to a certain extent, it relies too much on the accuracy and completeness of keywords and is difficult to process complex semantic information.

[0004] Furthermore, outsiders such as tourists are currently unable to understand the cultural assets of an institution, such as collections, in an efficient manner. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the present application is proposed to solve or at least partially solve the technical problems existing in the prior art.

[0006] On one hand, the present application provides a method for automatic classification of multimodal data, comprising:

[0007] Obtain vertical domain multimodal data to be classified;

[0008] Inputting the multimodal data into a vertical domain model to generate cross-modal labels corresponding to the multimodal data, wherein the vertical domain model is obtained based on a base large model;

[0009] Associating the cross-modal tags to a knowledge graph.

[0010] In one embodiment, obtaining the vertical domain model based on the base macro model includes:

[0011] The base large model is transformed to form a target large model, wherein the target large model sets a text encoder, an image encoder, an audio encoder and a video encoder at the original input layer of the base large model to extract features of samples of corresponding types; an input projector is set to project the extracted features into a common feature space; feature fusion is realized by using the interaction between the self-attention mechanism of the base large model and the feedforward network; a classifier is set after the original output layer of the base large model to generate cross-modal labels; a knowledge extractor is set after the classifier to extract entities and relationships from the labels;

[0012] Pre-training the target large model to obtain a pre-trained target large model;

[0013] The pre-trained target large model is fine-tuned to obtain the vertical domain model.

[0014] In one embodiment, pre-training the target large model to obtain the pre-trained target large model includes:

[0015] Acquire a large-scale multimodal data sample, wherein the large-scale multimodal data sample includes a large-scale text sample, a large-scale image sample, a large-scale audio sample, and a large-scale video sample;

[0016] The target large model is pre-trained based on the large-scale multimodal data sample to obtain the target large model.

[0017] In one embodiment, fine-tuning the pre-trained target large model to obtain the vertical domain model includes:

[0018] Acquire vertical domain related multimodal data samples, wherein the multimodal data samples include vertical domain text samples, vertical domain picture samples, vertical domain audio samples and vertical domain video samples;

[0019] The pre-trained target large model is fine-tuned based on the multimodal data samples to obtain the vertical domain model.

[0020] In one embodiment, fine-tuning the pre-trained target large model based on the multimodal data sample to obtain the vertical domain model includes:

[0021] Providing a transfer learning method library, wherein the method library includes multiple transfer learning methods;

[0022] Calculate the quantity ratio of different types of samples in the obtained vertical domain related multimodal data samples;

[0023] According to the type of the acquired vertical domain-related multimodal data samples, an appropriate transfer learning method is called from the method library to perform fine-tuning training on the pre-trained target large model to obtain the vertical domain large model.

[0024] In one embodiment, the method further comprises:

[0025] Define the entities, relationships and attributes of the knowledge graph according to the specification constraints of the vertical domain;

[0026] Associating the cross-modal tag with the knowledge graph includes:

[0027] Entity recognition and relationship extraction are performed by the knowledge extractor to obtain entities and relationships respectively;

[0028] Link the entities and relations to the knowledge graph.

[0029] In one embodiment, linking the entities and relationships to a knowledge graph includes:

[0030] Determine the degree of agreement between the triples of entities and relations output by the knowledge extractor and the existing triples in the current knowledge graph;

[0031] Based on the degree of fit, link to the current knowledge graph.

[0032] In one embodiment, linking to the current knowledge graph based on the degree of fit includes:

[0033] Select the triple consisting of entities and relations with the greatest degree of fit;

[0034] Inputting the triples consisting of the entities and relations with the greatest degree of fit back into the vertical domain big model to perform common sense detection that conforms to the vertical domain;

[0035] If the detection result of the vertical domain large model is consistent with common sense, the new triple is retained to be integrated with the current knowledge graph.

[0036] In one embodiment, the method further comprises:

[0037] If it is detected that the meaning of the cross-modal label has different semantics, the semantics corresponding to the multi-semantic label is outputted through the vertical domain large model;

[0038] A multi-semantic mapping relationship is formed between the multi-semantic tag and the corresponding multiple semantics, and is stored in a multi-semantic mapping database.

[0039] A second aspect of the present application provides a multimodal data query method, comprising:

[0040] Presenting a query entry and a list of entity names of the knowledge graph obtained by the method of the first aspect on the first human-computer interaction interface;

[0041] In response to an entity name input by a first user at the query entry, presenting a local knowledge graph having an edge relationship with the entity name on the human-computer exchange interface;

[0042] In response to the first user clicking on an entity in the local knowledge graph, the corresponding multimodal data is presented on the human-computer interaction interface in the form of thumbnails.

[0043] The third aspect of the present application provides a vertical domain knowledge question answering method, including:

[0044] Presenting a question-and-answer entrance on the second human-computer interaction interface;

[0045] Receiving a question input by a second user in the question-and-answer portal;

[0046] The answer corresponding to the question is output at the question-and-answer entrance at least based on the vertical domain big model and the knowledge graph, wherein the vertical domain big model and the knowledge graph are obtained according to the method described in the first aspect of the present application.

[0047] In one embodiment, outputting the answer corresponding to the question at the question-and-answer entrance based at least on the vertical domain big model and the knowledge graph includes:

[0048] If the user's query intent is identified through a vertical domain big model, and the entity in the question is determined;

[0049] Link the entities identified in the question with entities in the knowledge graph;

[0050] Check whether the entity in the linked knowledge graph has the identifier.

[0051] If the entity in the linked knowledge graph has no identifier, construct a knowledge graph query statement according to the question intent and the linked entity;

[0052] If the entity in the linked knowledge graph has an identifier, construct a knowledge graph query statement and a multi-semantic mapping database query statement according to the question intent and the linked entity;

[0053] Use the constructed knowledge graph query statement and multi-semantic mapping database query statement to retrieve relevant information in the knowledge graph and multi-semantic mapping database respectively;

[0054] Based on the retrieved relevant information, the answer corresponding to the question is output at the question and answer entrance.

[0055] The fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory, wherein the memory stores computer-executable instructions, and the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the classification method described in the first aspect, the query method described in the second aspect, or the question-and-answer method described in the third aspect.

[0056] The fifth aspect of the present application provides a computer-readable storage medium, in which multiple program codes are stored, and the program codes are suitable for being loaded and run by a processor to execute the classification method described in the first aspect, or the query method described in the second aspect, or the question-and-answer method described in the third aspect.

[0057] The above one or more technical solutions of the present application have at least one or more of the following beneficial effects:

[0058] Through the classification method of the present application, massive multimodal data can be classified efficiently;

[0059] Through the query method of the present application, files can be efficiently queried and managed.

[0060] Through the question-answering method of the present application, knowledge in a certain field can be understood efficiently and comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0062] Figure 1 is a schematic diagram of a hardware architecture for implementing a method of the present application according to an embodiment of the present application;

[0063] Figure 2 This is a network framework diagram of a vertical domain large model according to an embodiment of the present application;

[0064] Figure 3 It is a schematic flow chart of the main steps of a method for automatic classification of multimodal data according to an embodiment of the present application;

[0065] Figure 4 It is a flowchart of the main steps of a multimodal data query method according to an embodiment of the present application;

[0066] Figure 5 A flowchart of the main steps of a vertical domain knowledge question answering method according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] Some embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application and are not intended to limit the protection scope of the present application.

[0068] In the description of this application, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Computer-readable storage media include any suitable media that can store program codes, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc.

[0069] The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one of A or B" or "at least one of A and B" has a similar meaning to "A and / or B" and may include only A, only B, or A and B. The singular forms "a", "the" and "the" may also include plural forms.

[0070] Here we first explain some terms involved in this application.

[0071] Large language model: refers to a model trained using deep learning technology that can process large-scale data.

[0072] Multimodal large language model: Able to process and understand data of multiple different modalities at the same time. Multimodal large language model can use data of multiple modalities for joint modeling and prediction at the same time, so as to understand and process the input information more comprehensively.

[0073] Vertical Domain: refers to the requirements and characteristics for a specific industry or application field. These requirements are usually industry-specific, such as medical, financial, telecommunications, etc. Software architecture solutions in vertical domains are usually designed to meet the business process and regulatory requirements of a specific industry.

[0074] Base large model: refers to a general large language model that has not undergone additional instruction fine-tuning or downstream task adaptation fine-tuning.

[0075] Vertical domain big model: refers to the "special model" for a specific field obtained by fine-tuning or adapting the base big model using data samples from a specific field.

[0076] Knowledge graph: refers to a technical method that uses a graph model to describe knowledge and model the relationship between all things in the world. It consists of nodes and edges. Among them, nodes represent entities, concepts or attribute values, edges represent relationships, and entities and relationships form triples.

[0077] Multimodal knowledge graph: refers to the presentation of knowledge in a way that expands a single text-based node into multimodal data including digital images, audio, etc. Multimodal data is integrated with the corresponding entities, relationships, and attributes of the text to complete the construction of a multimodal knowledge graph.

[0078] Figure 1 A structural block diagram of a computer system provided by an exemplary embodiment of the present application is shown, and the method involved in the present application is executed by the computer system or part of the computer system.

[0079] The computer system includes: a first server 12, a first terminal 14, a second server 10, and a second terminal 17. The first terminal 14 is connected to the first server 12 via a wireless network or a wired network. The first server is connected to the remote second server 10 via a wireless network. The second terminal 17 is also connected to the remote second server 10 via a wireless network.

[0080] The first server 12 and the second server 10 may each include at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. The server includes a processor and a memory.

[0081] The first terminal 14 and the second terminal 17 can be desktop computers or portable mobile terminals, such as smart phones, tablet computers, laptop computers, etc., as long as they have display and input functions. The first terminal 14 and the second terminal 17 also include a processor and a memory.

[0082] Among them, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to process computing operations such as large model modeling and training.

[0083] The memory may include one or more computer-readable storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory is used to store at least one instruction, which is used to be executed by the corresponding processor to implement the method involved in the present application.

[0084] In a specific application scenario, a model developer, a model demander, and an external party are involved. Among them, the model demander is the party that needs to classify (i.e. label, these two terms are equivalent in this application) and manage the complex data of the unit, such as the cultural unit mentioned in the background technology part. The various multimodal data of the unit may be distributed in the computers of the employees 16. The second server 10 is the server of the model developer, which has strong computing power. For example, it has an AI processor, which is used to perform secondary development based on the existing open source base model, build a new network architecture for downstream tasks, and then use vertical domain data samples (such as cultural field data samples) to train the new network architecture to obtain the vertical domain large model of this application. Finally, the vertical domain large model is used to classify the various multimodal data of the demander unit to facilitate the management of the demander.

[0085] Optionally, the vertical domain large model can still be deployed in the second server 10 after training, and various multimodal data of the demand-side unit can be uploaded to the first server 10 through the first server 12 for classification. The employee 16 can perform query management through the relevant management software on his first terminal 14, for example, through the account assigned to him.

[0086] Alternatively, after training, the vertical domain large model can be deployed in the demand-side unit's own server, that is, in the first server 12. The advantage of this is better confidentiality.

[0087] Alternatively, if the computing power of the demand side's own server is good, the work of secondary development, construction and training of a new network architecture for downstream tasks based on the existing open source base model can also be completed on its own server. In this case, the participation of the second server 10 is not required.

[0088] On the other hand, after classifying the data of the demand-side unit, the present application also provides a question-and-answer system for external users 18 to query and understand the cultural assets of the demand-side unit, and the user 18 can understand it through his own terminal device 17. Alternatively, the demand-side unit can develop a cultural asset display all-in-one machine with a server and display and input functions, such as placing it in an exhibition hall, for external users 18 to query. In this case, the terminal device 17 is not needed.

[0089] Generate vertical domain model

[0090] For the classification (i.e. labeling) of multimodal data, the existing general large language model is not satisfactory, but the large language model has a strong semantic understanding ability. For this reason, the inventors chose an open source base large model for secondary development, such as using ChatGLM as the base large model. ChatGLM is an open language model based on the general language model GLM framework.

[0091] In order to meet the needs and facilitate the classification query of one's own multimodal data, this application builds a new network architecture based on the selected base large model, see the attached Figure 2 , Figure 2 This is a network architecture according to an embodiment of the present application.

[0092] The ChatGLM model is designed based on the Transformer architecture and includes an encoder and a decoder. ChatGLM is mainly used to process natural language tasks. Its core advantage lies in understanding and generating text rather than directly processing multimodal data. Therefore, in order to process multimodal feature extraction, as shown in the figure, an additional encoder module is added to the original input layer 20 of the ChatGLM model, including a text encoder 21, an image encoder 22, an audio encoder 23, and a video encoder 24.

[0093] Among them, the text encoder 21, such as the BERT model, is used to extract the semantic features of text data; the image encoder 22, such as the CNN model, is used to extract the visual features of picture data; the audio encoder 22, such as Wave2Vec, is used to extract the audio features of audio data, such as Mel-frequency cepstral coefficients (MFCC); the video encoder 24, such as a two-stream network (Two-Stream Networks), is used to extract the visual and motion features of video data.

[0094] The extracted features of each modality need to be projected into a common feature space for effective fusion. To this end, an input projector 25 is added to map the features of different modalities into a representation aligned with the text feature space, which can be achieved through a linear layer, MLP.

[0095] Feature fusion is achieved by using the interaction between the ChatGLM model’s own self-attention mechanism and the feedforward network, which is indicated by reference numeral 26 in the figure. The self-attention mechanism allows the model to dynamically assign different weights between different words, thereby achieving feature fusion. The feedforward network further performs nonlinear transformations on these fused features to enhance the model’s expressive power.

[0096] The output layer 27 of the ChatGLM model generates a log probability distribution, which is a vector of raw scores for all possible categories. A classifier 28 is added after the output layer 27, for example, a softmax function is applied to the output layer. The Softmax function converts these raw scores into a probability distribution so that the sum of the probabilities of all categories is 1. In this way, each category has a predicted probability between 0 and 1. Based on the output of the classifier 28, the category with the highest probability is selected as the final classification result. This step is usually implemented by the argmax function, which returns the index of the category with the highest probability.

[0097] At this point, the data of each modality is labeled with semantic labels, which have the nature of "cross-modality". This is because due to the existence of encoders for each modality of data and subsequent feature fusion, the data features of different modalities are comprehensively considered, making the generated cross-modal labels more accurate. In other words, in this process, data features of different modalities such as text, audio, pictures, and videos are integrated into a unified feature space, influencing each other and jointly determining the final cross-modal label.

[0098] For example, when generating cross-modal tags for text, the constructed model will comprehensively consider the features of the text and the features of audio, pictures, and videos. This allows the cross-modal tags of text to contain not only the information of the text itself, but also the relevant information of multi-modal data such as audio, pictures, and videos. Similarly, the generation of cross-modal tags for audio and video will also be affected by the features of other modal data.

[0099] The granularity and form of the labels for each modal data may be different. For example, for the text modality, the cross-modal labels generated are usually regarded as one or more words, because the main function of text is to express meaning, and usually only a concise and clear word or phrase is needed to convey the information. For audio or video modalities, the generated cross-modal labels may be more inclined to the phrase or sentence level, because the data of these modalities usually contain a large number of audio frames or video frames, so longer sequences can be extracted from them to generate corresponding cross-modal labels. Of course, this is not absolute. In some scenarios, such as when complex audio or video content needs to be classified, longer or more complex label sequences may be generated. At the same time, for various modal data, the granularity and form of the labels can also be adjusted according to actual needs and application scenarios.

[0100] In order to facilitate query, this application hopes to associate the cross-modal tags of each multimodal data with the knowledge graph 30. In the ChatGLM network architecture, the ChatGLM model is a unidirectional language model, which is mainly used to predict the next word, rather than to perform entity recognition and relationship extraction for a given tag. Therefore, it is necessary to combine other technologies and methods to achieve this. Figure 2 A knowledge extractor 29 is introduced into the network architecture.

[0101] In one example, LSTM (Long Short-Term Memory) is used to act as a knowledge extractor. LSTM is a common sequence model that can be used to process sequence data. Since what needs to be processed is a label (a word or sentence, essentially a text sequence), LSTM is suitable. The LSTM model can be used to model entity recognition and relationship extraction problems in text sequences. Specifically, we can regard the text sequence as a sequence data and use LSTM to capture long-term dependencies in the sequence.

[0102] In the ChatGLM network architecture, we do not need to use LSTM at every position, but can only use it at the specific position of interest. In other words, an additional LSTM module can be added outside the output layer of the ChatGLM model to model specific entity recognition and relationship extraction tasks. This module can share part of the ChatGLM model, thereby reducing the number of model parameters and computational complexity.

[0103] Next, the newly constructed network architecture needs to be trained to eventually obtain a large vertical domain model.

[0104] The training of the newly constructed network architecture (hereinafter referred to as the target large model) generally includes two processes: S10 pre-training and S20 fine-tuning training.

[0105] Specifically, the steps of S10 pre-training include:

[0106] S100. Acquire a large-scale multimodal data sample, wherein the large-scale multimodal data sample includes a large-scale text sample, a large-scale picture sample, a large-scale audio sample, and a large-scale video sample.

[0107] These large-scale samples are manually annotated samples and can be obtained through the Internet or commercial databases.

[0108] S105. Pre-train the target large model based on the large-scale multimodal data samples to obtain a pre-trained target large model.

[0109] The original multimodal data samples need to be converted into a representation suitable for processing in ChatGLM. For data in different modalities such as text, audio, and video, corresponding preprocessing techniques can be used for processing.

[0110] The loss function of the training uses a cross entropy loss function. In one example, the training stops when the loss is less than a preset threshold.

[0111] The steps of S20 fine-tuning training include:

[0112] S200: Acquire vertical domain related multimodal data samples, wherein the multimodal data samples include vertical domain text samples, vertical domain picture samples, vertical domain audio samples, and vertical domain video samples.

[0113] In one embodiment of the present application, the vertical domain is the field of cultural industry.

[0114] The vertical domain-related multimodal data samples may be provided by demand-side customers in the relevant fields.

[0115] S205. Fine-tune the pre-trained target large model based on the multimodal data samples to obtain the vertical domain model.

[0116] In one embodiment, step S205 further includes:

[0117] S2050. Provide a transfer learning method library, wherein the method library includes multiple transfer learning methods.

[0118] For example, the strategy library includes P-Tuning method, LoRA method, full parameter training method, Prompt Tuning method, Feature Adaption method, etc.

[0119] S2052. Calculate the quantity ratio of different types of samples in the obtained vertical domain related multimodal data samples.

[0120] S2054. Call an appropriate transfer learning method from the method library according to the type of the acquired vertical domain-related multimodal data samples to fine-tune the pre-trained target large model to obtain the vertical domain large model.

[0121] For example, among the vertical domain related multimodal data samples obtained, image samples are the most numerous. In this case, the full parameter training method is selected to fine-tune the image encoder part to make full use of a large amount of data to improve the performance of the model in image recognition tasks.

[0122] For example, if the number of text samples is moderate, the Prompt Tuning method is selected to optimize the prompt to adapt to the specific text understanding task. This method is parameter-efficient and suitable for medium-sized data.

[0123] For example, if the number of audio texts is small, the LoRA method or the P-Tuning method can be selected. These methods have advantages in parameter efficiency, are suitable for fine-tuning of small-scale data, and can reduce the risk of overfitting.

[0124] Generally speaking, in the prior art of fine-tuning the model using the transfer learning method, a fixed transfer learning method is used. However, this application fully pays attention to the fact that multimodal samples are provided by various demand-side units, but in practice, it is found that the proportion of multimodal samples that can be provided by various demand-side units is not the same, some have more video samples, and some have more text samples. In this case, if a transfer learning method is used, the obtained vertical model will not be targeted, affecting the customer experience.

[0125] Additionally, in one embodiment, Figure 2 The network architecture can also deploy a reward model for training in a RLHF reinforcement learning manner to fine-tune some parameters of the target large model mentioned above. RLHF allows humans to directly guide machines and allows machines to master decision-making elements that are clearly embedded in human experience. As an effective alignment technique, RLHF can alleviate the harmful content generated by large models to a certain extent and improve information integrity.

[0126] Defining the Knowledge Graph

[0127] To design a knowledge graph, it is necessary to define entities, relationships, and attributes based on the requirements of the demander (often subject to the norms of the field, such as common classification patterns, etc.).

[0128] Entity definition: can be used as a node type in the knowledge graph.

[0129] Relationship definition: defines the relationship between entities.

[0130] · Attribute definition: Add attributes to the entity.

[0131] Still taking the museum as an example, we design the knowledge graph:

[0132] 1. Design Entities

[0133] Cultural Relics: The main collections in a museum.

[0134] Historical Event: The historical background related to the cultural relics.

[0135] Person: A historical figure or donor related to the cultural relic.

[0136] Exhibition: Exhibitions held by museums.

[0137] Document: research and description documents about cultural relics.

[0138] 2. Design relationships

[0139] Belongs to: The relationship between the artifact and the historical period to which it belongs.

[0140] Participated in: The relationship between a person and historical events.

[0141] · Created: The creative relationship between the characters and the artifacts.

[0142] Displayed in: The relationship between the artifact and the exhibition.

[0143] Recorded in: The information of cultural relics is recorded in documentary materials.

[0144] At this point, customers can use the vertical domain model to classify and label the multimodal data to be classified and associate it with the knowledge graph. Next, we will move on to the description of the model application phase.

[0145] Automatic classification (labeling) of multimodal data

[0146] In one embodiment, Figure 3 As shown, the multimodal data automatic classification method of the present application includes:

[0147] S300: Obtain vertical domain multimodal data to be classified.

[0148] The data to be classified is what the demander needs to label, including at least two of text data, image data, audio data and video data.

[0149] In one embodiment, for example, an employee has put multiple types of files with certain relevance in a folder, and uploads data from the folder.

[0150] In one example, the acquired vertical domain multimodal data includes:

[0151] Textual material: "Along the River During the Qingming Festival was painted by Zhang Zeduan during the Northern Song Dynasty."

[0152] Picture information: High-definition picture of "Along the River During the Qingming Festival".

[0153] Audio material: "Along the River During the Qingming Festival is now in the collection of the Palace Museum in Beijing."

[0154] Video material: A documentary introducing "Along the River During the Qingming Festival".

[0155] S305: Input the multimodal data into a vertical domain model to generate cross-modal labels corresponding to each of the multimodal data.

[0156] Continuing with the previous example, we get the following cross-modal tags:

[0157] Text tags: "Northern Song Dynasty", "Zhang Zeduan", "Along the River During the Qingming Festival";

[0158] Image tag: "Along the River During the Qingming Festival";

[0159] Audio tags: "Along the River During the Qingming Festival", "Beijing Palace Museum";

[0160] Video tags: "Along the River During the Qingming Festival", "Documentary".

[0161] S310: Associating the cross-modal tag with the knowledge graph.

[0162] In one embodiment, step S310 specifically includes:

[0163] S3100, performing entity recognition and relationship extraction through the knowledge extractor to obtain entities and relationships respectively.

[0164] Still continuing with the previous example, entity recognition:

[0165] The cultural relic “Along the River During the Qingming Festival”, the person “Zhang Zeduan” and the historical period “Northern Song Dynasty” were identified from the textual data.

[0166] Identify the cultural relic "Along the River During the Qingming Festival" from the picture materials.

[0167] The cultural relic "Along the River During the Qingming Festival" and the institution "Beijing Palace Museum" were identified from the audio material.

[0168] Identify the cultural relic "Along the River During the Qingming Festival" and related historical background from video footage.

[0169] Relation Extraction

[0170] According to the text data, the relations "Along the River During the Qingming Festival belongs to the Northern Song Dynasty" and "Zhang Zeduan created Along the River During the Qingming Festival" are extracted.

[0171] According to the audio data, the relationship "Along the River During the Qingming Festival is exhibited in the Palace Museum in Beijing" is extracted.

[0172] According to the video data, the relationship "Along the River During the Qingming Festival is recorded in the documentary" is extracted.

[0173] S3105. Link the entities and relationships to the knowledge graph.

[0174] S31050, determining the degree of agreement between the triples consisting of entities and relations output by the knowledge extractor and the triples already in the current knowledge graph;

[0175] The degree of agreement is determined by cosine similarity.

[0176] S31055. Link to the current knowledge graph based on the degree of fit.

[0177] In one embodiment, if the degree of fit exceeds a preset threshold, it is considered to be a fit, and the new data can be integrated with the data in the existing knowledge graph to enrich and expand the content of the knowledge graph. This helps to achieve information fusion and unified management between different data sources.

[0178] In another embodiment, in order to more accurately link to the current knowledge graph, instead of comparing with a fixed threshold, the following method is used:

[0179] Select the triple consisting of entities and relations with the greatest degree of fit;

[0180] Inputting the triples consisting of the entities and relations with the greatest degree of fit back into the vertical domain big model to perform common sense detection that conforms to the vertical domain;

[0181] If the detection result of the vertical domain big model is consistent with common sense, the new triple is retained to be integrated with the current knowledge graph. For example, Zhang Zeduan's creation of "Along the River During the Qingming Festival" is consistent with common sense judgment. The vertical domain big model has been trained with large-scale data samples and vertical domain samples in the early stage, so that it can better understand the common sense of the vertical domain, thereby better enriching and expanding the content of the knowledge graph.

[0182] Continuing with the above example, the following shows how the above example can be integrated into the knowledge graph.

[0183] Entity instantiation:

[0184] Cultural Relic Example: "Along the River During the Qingming Festival"

[0185] Character example: "Zhang Zeduan"

[0186] Historical period example: "Northern Song Dynasty"

[0187] Institutional example: "The Palace Museum, Beijing"

[0188] · Documentary example: "Along the River During the Qingming Festival" documentary

[0189] Relationship instantiation:

[0190] "Along the River During the Qingming Festival" belongs to the Northern Song Dynasty"

[0191] "Zhang Zeduan created Along the River During the Qingming Festival"

[0192] ·“Along the River During the Qingming Festival” is exhibited at the Palace Museum in Beijing”

[0193] "Along the River During the Qingming Festival is recorded in the documentary Along the River During the Qingming Festival."

[0194] It should be noted that in the above examples, the cross-modal tags and entities are consistent, and it seems that no entity recognition step is required and they can be used directly as entities. However, as mentioned earlier, for videos, cross-modal tags may be more inclined to the phrase or sentence level, so entity recognition and relationship extraction are required to divide them into independent subjects and objects.

[0195] The extracted entities and relationships between entities are stored in a graph database, such as Neo4j or OrientDB.

[0196] If the entities and relations extracted by the knowledge extractor do not always match the current knowledge graph, for example, one can consider improving the knowledge extractor to improve the accuracy and completeness of the knowledge graph.

[0197] Those skilled in the art will understand that the above-mentioned association action will be performed each time the data to be classified is uploaded so that the knowledge graph is continuously updated.

[0198] Multi-semantic mapping relationship

[0199] In the process of training the large model with samples from the cultural field, the inventors found that the meanings of some labels have different semantics in different contexts. For example, the semantics of the generated label "Spring and Autumn" include historical periods, seasons, and even book titles. This may cause inaccurate returned results when performing retrieval queries in the future. For this reason, the ChatGLM model of the present application can also output the semantics corresponding to labels with multiple semantics. Furthermore, the present application forms a multi-semantic mapping relationship between the multi-semantic label and the corresponding multiple semantics, and stores it in the multi-semantic mapping database 31, such as Figure 2 As shown. At the same time, the knowledge graph entity corresponding to the multi-semantic label is marked. The role of the multi-semantic mapping database 31 will be further introduced later when it comes to questions and answers for external personnel.

[0200] Query and management of multimodal data

[0201] After the above knowledge graph is constructed, the demand-side staff is provided with the ability to efficiently query and manage files.

[0202] To this end, a query software system is developed, and employees 16 can log in to their own accounts to query. Specifically, an embodiment of the present application provides a multimodal data query method, such as Figure 4 As shown, including:

[0203] S400, presenting a query entry and a list of entity names of the above-mentioned knowledge graph on a human-computer interaction interface;

[0204] S405, in response to the entity name input by the employee at the query entrance, presenting a local knowledge graph having an edge relationship with the entity name on the human-computer exchange interface;

[0205] S410: In response to an employee clicking on an entity in the local knowledge graph, presenting corresponding multimodal data on the human-computer interaction interface in the form of thumbnails.

[0206] The user can further click on the thumbnail to generate a preview. Further, if the user is interested, they can also download it if their personal permissions allow.

[0207] Trivia

[0208] In order to allow external personnel such as tourists to better understand the knowledge of vertical domains, a vertical domain (such as cultural domain) knowledge question and answer system has been developed. User 18 can log in to the system through the second terminal 17 or the aforementioned integrated machine to ask questions.

[0209] Specifically, an embodiment of the present application provides a vertical domain knowledge question answering method, such as Figure 5 As shown, including:

[0210] S500, presenting a question-and-answer entrance on the human-computer interaction interface;

[0211] S505, receiving a question input by a questioner in the question-and-answer portal;

[0212] S510: Output the answer corresponding to the question at the question-and-answer entrance based at least on the vertical domain big model and the knowledge graph.

[0213] In one embodiment, S510 further includes:

[0214] S5100: If the user's query intention is identified through the vertical domain big model, and the entity in the question is determined;

[0215] S5105, linking the entity identified in the question with the entity in the knowledge graph;

[0216] S5110. Detect whether the entity in the linked knowledge graph has the identifier.

[0217] S5115. If the entity in the linked knowledge graph has no identifier, construct a knowledge graph query statement according to the question intent and the linked entity;

[0218] S5120: If the entity in the linked knowledge graph has an identifier, construct a knowledge graph query statement and a multi-semantic mapping database query statement according to the question intent and the linked entity;

[0219] S5125, using the constructed knowledge graph query statement and multi-semantic mapping database query statement to retrieve relevant information in the knowledge graph and multi-semantic mapping database respectively;

[0220] S5130. Output the answer corresponding to the question at the question and answer entry according to the retrieved relevant information.

[0221] Queries in the knowledge graph can be implemented through operations such as graph traversal and pattern matching;

[0222] Queries in the multi-semantic mapping database can be implemented through operations such as vector Euclidean similarity.

[0223] In this embodiment, it is important to distinguish whether the keyword in the question has multiple semantics in the entity in the knowledge graph. If there are multiple semantics, there will be misjudgment by simply searching the knowledge graph. For example, the question asked by the user is "I want to know what clothes to wear in Spring and Autumn". In this sentence, Spring and Autumn may have two semantics, one is the historical period, and the other is the season. At this time, the mapping relationship between the entity and the semantics must be called for judgment.

[0224] It is understandable to those skilled in the art that the present application implements all or part of the processes in the classification method, query method or question-answering method of the above-mentioned embodiments, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electrical carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0225] The present application also provides an electronic device. Figure 1 The server 12 shown can be used to execute the above classification method; for example Figure 1 The first terminal 14 shown can be used to execute the above query method; for example Figure 1 The second terminal 18 in the embodiment can be used to execute the above question-answering method.

[0226] It should be pointed out that although the various steps in the above embodiments are described in a specific order, those skilled in the art can understand that in order to achieve the effect of the present application, different steps do not have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders. These adjusted schemes are equivalent to the technical schemes described in this application, and therefore will also fall within the scope of protection of this application.

Claims

1. A method for automatic classification of multimodal data, characterized in that: include: Obtain vertical domain multimodal data to be classified; Inputting the multimodal data into a vertical domain model to generate cross-modal labels corresponding to the multimodal data, wherein the vertical domain model is obtained based on a base large model; Associating the cross-modal tags to a knowledge graph.

2. The method according to claim 1, characterized in that: The step of obtaining the vertical domain model based on the large base model includes: The base large model is transformed to form a target large model, wherein the target large model sets a text encoder, an image encoder, an audio encoder and a video encoder at the original input layer of the base large model to extract features of samples of corresponding types; an input projector is set to project the extracted features into a common feature space; feature fusion is realized by using the interaction between the self-attention mechanism of the base large model and the feedforward network; a classifier is set after the original output layer of the base large model to generate cross-modal labels; a knowledge extractor is set after the classifier to extract entities and relationships from the labels; Pre-training the target large model to obtain a pre-trained target large model; Fine-tune the pre-trained target large model to obtain the vertical domain model.

3. The method according to claim 2, characterized in that The pre-training of the target large model to obtain the pre-trained target large model includes: Acquire a large-scale multimodal data sample, wherein the large-scale multimodal data sample includes a large-scale text sample, a large-scale image sample, a large-scale audio sample, and a large-scale video sample; The target large model is pre-trained based on the large-scale multimodal data sample to obtain the target large model.

4. The method according to claim 3, characterized in that The fine-tuning training of the pre-trained target large model to obtain the vertical domain model includes: Acquire vertical domain related multimodal data samples, wherein the multimodal data samples include vertical domain text samples, vertical domain picture samples, vertical domain audio samples and vertical domain video samples; The pre-trained target large model is fine-tuned based on the multimodal data samples to obtain the vertical domain model.

5. The method according to claim 4, characterized in that The fine-tuning training of the pre-trained target large model based on the multimodal data sample to obtain the vertical domain model includes: Providing a transfer learning method library, wherein the method library includes multiple transfer learning methods; Calculate the quantity ratio of different types of samples in the obtained vertical domain related multimodal data samples; According to the type of the acquired vertical domain-related multimodal data samples, an appropriate transfer learning method is called from the method library to perform fine-tuning training on the pre-trained target large model to obtain the vertical domain large model.

6. The method according to any one of claims 2 to 5, characterized in that: The method further comprises: Define the entities, relationships and attributes of the knowledge graph according to the specification constraints of the vertical domain; Associating the cross-modal tag with the knowledge graph includes: Entity recognition and relationship extraction are performed by the knowledge extractor to obtain entities and relationships respectively; Link the entities and relations to the knowledge graph.

7. A multimodal data query method, characterized in that: include: Presenting a query entry and a list of entity names of the knowledge graph obtained by the method according to any one of claims 1 to 6 on the first human-computer interaction interface; In response to an entity name input by a first user at the query entry, presenting a local knowledge graph having an edge relationship with the entity name on the human-computer exchange interface; In response to the first user clicking on an entity in the local knowledge graph, the corresponding multimodal data is presented on the human-computer interaction interface in the form of thumbnails.

8. A vertical domain knowledge question answering method, characterized in that: include: Presenting a question-and-answer entrance on the second human-computer interaction interface; Receiving a question input by a second user in the question-and-answer portal; The answer corresponding to the question is output at the question-and-answer entrance at least based on the vertical domain big model and the knowledge graph, wherein the vertical domain big model and the knowledge graph are obtained according to any one of the methods of claims 1-6.

9. An electronic device, characterized in that: It includes at least one processor and a memory, wherein the memory stores computer-executable instructions, and the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the classification method described in any one of claims 1 to 6, or the query method described in claim 7, or the question-answering method described in claim 8.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the classification method according to any one of claims 1 to 6, or the query method according to claim 7, or the question-answering method according to claim 8.

Images