Digital culture field abstract generation method, electronic equipment and storage medium

Abstract

The invention relates to a digital culture field abstract generation method, electronic equipment and a storage medium, and particularly provides a digital culture field abstract generation method which comprises the following steps: acquiring vertical domain multi-modal data of a to-be-generated abstract; generating a first cross-modal label set corresponding to the multi-modal data; associating the first cross-modal label set to a digital culture domain knowledge graph; forming a conceptual entity set based on the digital culture domain knowledge graph; expanding the concept entity set according to a specified threshold value and granularity to obtain an expanded entity set, and updating the digital culture domain knowledge graph based on the expanded entity set; obtaining a second cross-modal label set based on the vertical domain multi-modal data, comparing the expanded entity set with the second cross-modal label set, and sorting the expanded entity set to obtain a sorted entity set; and generating an abstract text based on the sorted entity set and the updated digital culture domain knowledge graph.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method for generating abstracts in the field of digital culture, an electronic device and a storage medium. Background Art

[0002] In the digital culture industry and other industries, the data uploaded by users are usually raw data that has not been deeply processed, and often lack detailed descriptions, background information, and contextual associations. These raw data are usually difficult to use directly during analysis, display, and storage, and cannot fully tap the potential value of the data. Currently, manual summarization or template application is generally used. For this type of summary generation method, the main problems faced by users are:

[0003] (1) Manual operation is cumbersome: Manual extension of description is time-consuming, especially when dealing with large-scale data sets. It is inefficient and requires a lot of human resources.

[0004] (2) Limitations of templated tools: Although templated tools have improved some efficiency, the descriptions they generate are often too simple and cannot provide targeted descriptions based on the specific content of the data. They lack flexibility and depth.

[0005] (3) Poor description consistency: Due to manual operation or template limitations, different batches of data may obtain inconsistent descriptions, affecting the accuracy of subsequent analysis and presentation.

[0006] (4) Insufficient intelligence: Traditional tools cannot conduct in-depth analysis based on the semantics, background, and context of the data, resulting in limited depth and breadth of descriptive information. Therefore, data expansion description becomes the key to solving this problem. By using AI (artificial intelligence) technology to expand the description of the data uploaded by users, descriptive content with rich semantics and detailed background can be automatically generated to help users better understand and use the data.

[0007] It can be seen that the current digital cultural field needs an AI system that can automatically expand the data uploaded by users. The system should be able to generate a comprehensive description based on the original data, including the background, characteristics, possible uses and related information of the data. Users expect that through this method, the time and workload of manual editing can be reduced, and the comprehensibility, usability and display effect of the data can be improved. Summary of the invention

[0008] In order to overcome the above defects, this application is proposed, that is, an intelligent summary generation method, which can perform multi-level recursive query based on folder hierarchy, file content and tags to achieve efficient and accurate data retrieval. Users can quickly find the required data in a complex file structure through simple query operations, and can perform detailed classification and intelligent retrieval based on file content and tag information.

[0009] Specifically, the first aspect of the present application provides a method for generating a summary in the field of digital culture, comprising:

[0010] Obtaining vertical domain multimodal data for which summaries are to be generated;

[0011] Inputting the multimodal data into a vertical domain model to generate a first cross-modal label set corresponding to each of the multimodal data, wherein the vertical domain model is obtained based on a base large model;

[0012] Associating the first cross-modal tag set to a digital culture domain knowledge graph;

[0013] Based on the digital culture field knowledge graph, searching for concept entities associated with the first cross-modal tag set to form a concept entity set;

[0014] According to the specified threshold and granularity, based on the digital culture domain knowledge graph, the concept entity set is expanded to obtain an expanded entity set, and the digital culture domain knowledge graph is updated based on the expanded entity set;

[0015] Obtaining a second cross-modal label set based on the vertical domain multimodal data, comparing the expanded entity set with the second cross-modal label set, and sorting the expanded entity set to obtain a sorted entity set;

[0016] Based on the sorted entity set and the updated digital culture domain knowledge graph, a summary text is generated.

[0017] In one embodiment, obtaining the vertical domain model based on the base macro model includes:

[0018] The base large model is transformed to form a target large model, wherein the target large model sets a text encoder, an image encoder, an audio encoder and a video encoder at the original input layer of the base large model to extract features of samples of corresponding types; an input projector is set to project the extracted features into a common feature space; feature fusion is realized by using the interaction between the self-attention mechanism of the base large model and the feedforward network; a classifier is set after the original output layer of the base large model to generate cross-modal labels; a knowledge extractor is set after the classifier to extract entities and relationships from the labels;

[0019] Pre-training the target large model to obtain a pre-trained target large model;

[0020] Fine-tune the pre-trained target large model to obtain the vertical domain model.

[0021] In one embodiment, based on the digital culture domain knowledge graph, conceptual entities associated with the first cross-modal tag set are searched to form a conceptual entity set:

[0022] Performing conceptual entity recognition on the tags associated with the first cross-modal tag set by a knowledge extractor to obtain the conceptual entity;

[0023] By comparing similarities, determining whether the conceptual entity is similar to an entity in the knowledge graph in the digital culture field;

[0024] In the case where the conceptual entity is similar to an entity in the digital culture field knowledge graph, the conceptual entity is included in the conceptual entity set.

[0025] In one embodiment, according to a specified threshold and granularity, based on the digital culture domain knowledge graph, the concept entity set is expanded to obtain an expanded entity set, and the digital culture domain knowledge graph is updated based on the expanded entity set, including:

[0026] While forming the concept entity set, if the similarity between the concept entity and the entity in the digital culture field knowledge graph meets the specified threshold and granularity, the concept entity is incorporated into the concept entity set to obtain the expanded entity set.

[0027] In one embodiment, a second cross-modal label set is obtained based on the vertical domain multimodal data, the expanded entity set is compared with the second cross-modal label set, and the expanded entity set is sorted to obtain a sorted entity set, including:

[0028] Sampling the data perpendicular to the multimodal data to obtain a second cross-modal label set;

[0029] For each entity in the expanded entity set, calculating a weight of the entity in the second cross-modal tag set;

[0030] The expanded entity set is sorted according to the frequency of occurrence of the entity in the second cross-modal tag set to obtain a sorted entity set.

[0031] In one embodiment, generating summary text based on the sorted entity set and the updated digital culture domain knowledge graph includes:

[0032] According to the ranking in the sorted entity set, each entity in the sorted entity set is expanded according to the specified summary expansion granularity to obtain an entity expansion sequence;

[0033] For each entity in the entity expansion sequence, a corresponding relationship is extracted in the digital culture field knowledge graph, and a summary text is generated in combination with the entity expansion sequence.

[0034] In one embodiment, the method further comprises:

[0035] The summary text is machine translated to obtain a summary translation text in a specified language.

[0036] The second aspect of the present application provides a digital culture field query method, characterized by comprising:

[0037] Presenting a query entry on a human-computer interaction interface;

[0038] In response to a user inputting a multimodal data query request instruction at the query entry, obtaining a query result;

[0039] In response to a user clicking on the query result, obtaining a summary corresponding to the query result according to any one of the methods in the first aspect;

[0040] The summary is presented on the human-computer interaction interface in the form of thumbnails.

[0041] The third aspect of the present application provides an electronic device, comprising at least one processor and a memory, wherein the memory stores computer-executable instructions, and the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the digital culture field abstract generation method described in any one of the first aspects, or the query method described in the second aspect.

[0042] The fourth aspect of the present application provides a computer-readable storage medium, in which multiple program codes are stored, characterized in that the program codes are suitable for being loaded and run by a processor to execute the digital culture field abstract generation method described in any one of the first aspects, or the query method described in the second aspect.

[0043] It can be seen that compared with the prior art, this solution has the following beneficial effects:

[0044] (1) Efficient and automated processing: Through the artificial intelligence big model, the system can automatically expand the description of the data uploaded by the user, reducing the time and cost of manual operations, which is particularly suitable for processing large-scale data sets.

[0045] (2) Multimodal data support: This solution can process different types of data (text, images, audio, video, etc.) and generate customized extended descriptions based on specific data types, breaking through the limitations of traditional template tools.

[0046] (3) Semantic enrichment and context association: Through semantic analysis and context expansion, the system can generate in-depth expanded descriptions to help users obtain more valuable information from the data and improve their understanding and use of the data.

[0047] (4) Consistency and flexibility: The description automatically generated by the system ensures the uniformity of the description and avoids the inconsistency problems that may occur in manual operation. In addition, the system supports users to customize the level of detail of the configuration description, which improves flexibility.

[0048] (5) Intelligent association and value mining: Through the knowledge graph, the system can identify the potential association between the uploaded data and other data, and provide related background and application scenarios, so that the value of the data uploaded by users can be mined to a greater extent.

[0049] Through this technology, users can quickly generate high-quality expansion descriptions, reduce the complexity of data processing, and improve the efficiency and effectiveness of data display and use. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The disclosure of the present application will become easier to understand with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present application. Among them:

[0051] Figure 1 A flow chart of a method for generating a summary in the digital culture field according to an embodiment of the present invention is shown.

[0052] Figure 2 A schematic diagram of a knowledge graph in the field of digital culture according to an embodiment of the present invention is shown.

[0053] Figure 3 A data flow diagram according to an embodiment of the present application is shown.

[0054] Figure 4 A schematic diagram of particle size selection according to an embodiment of the present patent is shown. DETAILED DESCRIPTION

[0055] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0056] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0057] Here we first explain some terms involved in this application.

[0058] Large language model: refers to a model trained using deep learning technology that can process large-scale data.

[0059] Multimodal large language model: Able to process and understand data of multiple different modalities at the same time. Multimodal large language model can use data of multiple modalities for joint modeling and prediction at the same time, so as to understand and process the input information more comprehensively.

[0060] Vertical domain: refers to the requirements and characteristics of a specific industry or application field. These requirements are usually industry-specific, such as medical, financial, telecommunications, etc. Software architecture solutions in vertical domains are usually designed to meet the business process and regulatory requirements of a specific industry.

[0061] Base large model: refers to a general large language model that has not undergone additional instruction fine-tuning or downstream task adaptation fine-tuning.

[0062] Vertical domain big model: refers to the "special model" for a specific field obtained by fine-tuning or adapting the base big model using data samples from a specific field.

[0063] Knowledge graph: refers to a technical method that uses a graph model to describe knowledge and model the relationship between all things in the world. It consists of nodes and edges. Among them, nodes represent entities, concepts or attribute values, edges represent relationships, and entities and relationships form triples.

[0064] Knowledge graph in the field of digital culture: refers to the knowledge graph extracted from different file types such as text records, digital images, audio, video, etc. in the field of digital culture. By integrating corresponding entities, relationships and attributes, professional field knowledge graphs for different types of files are obtained.

[0065] Figure 2 A schematic diagram of a knowledge graph in the digital culture field according to an embodiment is shown.

[0066] like Figure 2 As shown in the figure, Dream of the Red Chamber 201, Cao Xueqin 202, Lin Daiyu 203, National Ballet of China 204, and Tianqiao Theater 205 are all entities. These entities can have multiple attributes. For example, the attributes of Cao Xueqin 202 may include Qing Dynasty, novelist, etc. The relationship between entities is as follows Figure 2 As shown, Cao Xueqin 202 created Dream of the Red Chamber 201, the Central Ballet Company 204 performed Dream of the Red Chamber 201 at the Tianqiao Theater 205, the protagonist of Dream of the Red Chamber 201 is Lin Daiyu 203, and so on.

[0067] It is worth noting that the field of digital culture is different from general fields and has many proprietary terms or connotation relationships. Therefore, in the process of constructing the knowledge graph in the field of digital culture, we can use dictionary files in this field to enrich the comprehensiveness of the knowledge graph in the field of digital culture, thereby improving the accuracy and efficiency of feature recognition and file search.

[0068] Specifically, the digital culture domain knowledge graph can be an already constructed proprietary domain knowledge graph, an incomplete proprietary domain knowledge graph, or even a brand new proprietary domain knowledge graph. Since the concepts, entities, and terms of a field are constantly evolving with the times, the digital culture domain knowledge graph cannot be fixed and needs to be constantly enriched and updated. Therefore, after labeling different types of data files, it is very likely that the identified labels cannot find corresponding entities in the digital culture domain knowledge graph. At this time, the entities, attributes, and relationships corresponding to the labels can be expanded to the digital culture domain knowledge graph.

[0069] In general, the main innovation of the present invention is different from the prior art in that: in the current prior art, generally, the concept entities in the document are directly extracted through the artificial intelligence model, and the summary text is generated through machine learning. In the process of machine learning, some concept entities may be omitted. To remedy this defect, in the present invention, after extracting the concept entities in the document, these extracted concept entities will be compared with the original document (or a sample of the original document) to expand the concept entities, thereby improving the accuracy and completeness of summary generation.

[0070] In a first aspect, the present invention provides a method for generating abstracts in the field of digital culture.

[0071] Figure 1 FIG. 4 is a flow chart showing a method for generating a digital culture domain summary according to an embodiment of the present invention. Figure 1 As shown, the summary generation method in the field of digital culture includes:

[0072] S101: Obtain vertical domain multimodal data for which a summary is to be generated.

[0073] Specifically, in the vertical domain multimodal data for which a summary is to be generated, the vertical domain, as mentioned above, can be the needs and characteristics of a specific industry or application field. For example, the digital culture field can form a vertical domain. Multimodal data refers to data with multiple modalities, for example, it can have one modality among text, image, audio, video, or multiple modalities of data. It can be understood that the multimodal data obtained from the user input in S101 can be just one of the above modalities, or it can be a combination of multiple modal data.

[0074] S102: Input the multimodal data into a vertical domain model to generate a cross-modal label set corresponding to each of the multimodal data, wherein the vertical domain model is obtained based on a base large model.

[0075] In one embodiment, obtaining the vertical domain model based on the base macro model includes:

[0076] The base large model is transformed to form a target large model, wherein the target large model sets a text encoder, an image encoder, an audio encoder and a video encoder at the original input layer of the base large model to extract features of samples of corresponding types; an input projector is set to project the extracted features into a common feature space; feature fusion is realized by using the interaction between the self-attention mechanism of the base large model and the feedforward network; a classifier is set after the original output layer of the base large model to generate cross-modal labels; a knowledge extractor is set after the classifier to extract entities and relationships from the labels;

[0077] Pre-training the target large model to obtain a pre-trained target large model;

[0078] Fine-tune the pre-trained target large model to obtain the vertical domain model.

[0079] The following describes the generation process of the vertical domain model:

[0080] For the classification of multimodal data, the existing general large language model is not satisfactory, but the large language model has a strong semantic understanding ability. For this reason, the inventor chose an open source base large model for secondary development, such as using ChatGLM as the base large model. ChatGLM is an open language model based on the general language model GLM framework.

[0081] In order to meet the needs and facilitate the classification query of one's own multimodal data, this application builds a new network architecture based on the selected base large model, see the attached Figure 3 , Figure 3 This is a network architecture according to an embodiment of the present application. The dotted box 2 is a new network topology constructed on the basis of the base large model in the present technical solution.

[0082] The ChatGLM model is designed based on the Transformer architecture and includes an encoder and a decoder. ChatGLM is mainly used to process natural language tasks. Its core advantage lies in understanding and generating text rather than directly processing multimodal data. Therefore, in order to process multimodal feature extraction, as shown in the figure, an additional encoder module is added to the original input layer 30 of the ChatGLM model, including a text encoder 31, an image encoder 32, an audio encoder 33, and a video encoder 34.

[0083] Among them, the text encoder 31, such as the BERT model, is used to extract the semantic features of text data; the image encoder 32, such as the CNN model, is used to extract the visual features of picture data; the audio encoder 32, such as Wave2Vec, is used to extract the audio features of audio data, such as Mel-frequency cepstral coefficients (MFCC); the video encoder 34, such as a two-stream network (Two-Stream Networks), is used to extract the visual and motion features of video data.

[0084] The extracted features of each modality need to be projected into a common feature space for effective fusion. To this end, an input projector 35 is added to map the features of different modalities into a representation aligned with the text feature space, which can be achieved through a linear layer, MLP.

[0085] Feature fusion is achieved by using the interaction between the ChatGLM model’s own self-attention mechanism and the feedforward network, which is indicated by reference numeral 36 in the figure. The self-attention mechanism allows the model to dynamically assign different weights between different words, thereby achieving feature fusion. The feedforward network further performs nonlinear transformation on these fused features to enhance the model’s expressive power.

[0086] The output layer 37 of the ChatGLM model generates a log probability distribution, which is a vector of raw scores for all possible categories. A classifier 38 is added after the output layer 37, for example, a softmax function is applied to the output layer. The Softmax function converts these raw scores into a probability distribution so that the sum of the probabilities of all categories is 1. In this way, each category has a predicted probability between 0 and 1. Based on the output of the classifier 38, the category with the highest probability is selected as the final classification result. This step is usually implemented by the argmax function, which returns the index of the category with the highest probability.

[0087] At this point, the data of each modality is labeled with semantic labels, which have the nature of "cross-modality". This is because due to the existence of encoders for each modality of data and subsequent feature fusion, the data features of different modalities are comprehensively considered, making the generated cross-modal labels more accurate. In other words, in this process, data features of different modalities such as text, audio, pictures, and videos are integrated into a unified feature space, influencing each other and jointly determining the final cross-modal label.

[0088] For example, when generating cross-modal tags for text, the constructed model will comprehensively consider the features of the text and the features of audio, pictures, and videos. This allows the cross-modal tags of text to contain not only the information of the text itself, but also the relevant information of multi-modal data such as audio, pictures, and videos. Similarly, the generation of cross-modal tags for audio and video will also be affected by the features of other modal data.

[0089] The granularity and form of the labels for each modal data may be different. For example, for the text modality, the cross-modal labels generated are usually regarded as one or more words, because the main function of text is to express meaning, and usually only a concise and clear word or phrase is needed to convey the information. For audio or video modalities, the generated cross-modal labels may be more inclined to the phrase or sentence level, because the data of these modalities usually contain a large number of audio frames or video frames, so longer sequences can be extracted from them to generate corresponding cross-modal labels. Of course, this is not absolute. In some scenarios, such as when complex audio or video content needs to be classified, longer or more complex label sequences may be generated. At the same time, for various modal data, the granularity and form of the labels can also be adjusted according to actual needs and application scenarios.

[0090] like Figure 3 As shown, in order to facilitate query, this application hopes to associate the cross-modal tags of each multimodal data with the knowledge graph 5. In the ChatGLM network architecture, the ChatGLM model is a unidirectional language model, which is mainly used to predict the next word, rather than to perform entity recognition and relationship extraction for a given tag. Therefore, it is necessary to combine other technologies and methods to achieve this. To this end, in Figure 3 A knowledge extractor is introduced into the network architecture39.

[0091] In one example, LSTM (Long Short-Term Memory) is used as a knowledge extractor39. LSTM is a common sequence model that can be used to process sequence data. Since what needs to be processed is a label (a word or sentence, which is essentially a text sequence), LSTM is suitable. The LSTM model can be used to model entity recognition and relationship extraction problems in text sequences. Specifically, we can regard the text sequence as a sequence data and use LSTM to capture the long-term dependencies in the sequence.

[0092] In the ChatGLM network architecture, we do not need to use LSTM at every position, but can only use it at the specific position of interest. In other words, an additional LSTM module can be added outside the output layer of the ChatGLM model to model specific entity recognition and relationship extraction tasks. This module can share part of the ChatGLM model, thereby reducing the number of model parameters and computational complexity.

[0093] Next, the newly constructed network architecture needs to be trained to eventually obtain a large vertical domain model.

[0094] It is worth noting that in the process of training the model using samples from the digital cultural field, some labels have different semantics in different contexts. For example, the semantics of the generated label "Cherish Spring" includes "cherish spring" and the name of a character. This may cause inaccurate returned results when performing retrieval queries in the future. To this end, the ChatGLM model of the present application can also output the semantics corresponding to labels with multiple semantics. Furthermore, the present application forms a multi-semantic mapping relationship between the multi-semantic label and the corresponding multiple semantics, and stores it in the multi-semantic mapping database 4, such as Figure 3 At the same time, the knowledge graph entities corresponding to the multi-semantic labels are marked.

[0095] S103: Associating the first cross-modal tag set to the digital culture domain knowledge graph.

[0096] Specifically, the digital culture domain knowledge graph in the initial state can be a blank knowledge graph, a knowledge graph under construction, or a relatively complete knowledge graph. In other words, with the continuous input of data, the digital culture domain knowledge graph will become more complete.

[0097] After the tags in the first cross-modal tag set are extracted, these tags in the set will be enriched in the digital culture field knowledge graph, making the entities, attributes, and relationships in the digital culture field knowledge graph more complete.

[0098] S104: Based on the knowledge graph in the digital culture field, search for concept entities associated with the first cross-modal tag set to form a concept entity set.

[0099] In one embodiment, based on the knowledge graph in the digital culture field, searching for concept entities associated with the first cross-modal tag set to form a concept entity set includes:

[0100] Performing concept entity recognition on the tags associated with the first cross-modal tag set through a knowledge extractor to obtain the concept entities;

[0101] Judging whether the concept entities are similar to the entities in the knowledge graph of the digital culture field through similarity comparison;

[0102] In the case where the concept entities are similar to the entities in the knowledge graph of the digital culture field, incorporating the concept entities into the concept entity set.

[0103] Combined Figure 3 It can be seen that for each tag in the first cross-modal tag set, through the knowledge extractor 39, the concept entities corresponding to each tag are identified. For example, for a tag "Dream of the Red Chamber" in the first cross-modal tag set, through the knowledge extractor 39 to perform concept extraction on "Dream of the Red Chamber", a concept entity set such as {Dream of the Red Chamber, Cao Xueqin, Ming and Qing Dynasties novels} can be obtained. Comparing each concept entity in {Dream of the Red Chamber, Cao Xueqin, Ming and Qing Dynasties novels} with the entities in the knowledge graph of the digital culture field, for example, it is found that the entity "Gao E" in the knowledge graph of the digital culture field is similar to {Dream of the Red Chamber, Cao Xueqin, Ming and Qing Dynasties novels}, meeting the specified similarity judgment criteria, then "Gao E" can be expanded into the concept entity set {Dream of the Red Chamber, Cao Xueqin, Ming and Qing Dynasties novels}.

[0104] For the judgment of similarity, common similarity judgment methods in machine learning can be used, such as cosine similarity, classification tree, neural network and other methods.

[0105] It should be noted that when judging similarity, attention can also be paid to whether the entity has multiple semantics. If there are multiple semantics, there may be misjudgment. Therefore, when comparing similarity, with the help of, for example, Figure 3 the multi-semantic mapping database 4 in to distinguish between different semantics of the entity. For example, the entity is "Xi Chun". The entity "Xi Chun" has two semantics. One is "cherish spring", and the other is a character in Dream of the Red Chamber. At this time, it is necessary to use the multi-semantic mapping database 4 to call the mapping relationship between the entity and the semantics for judgment.

[0106] S105: According to the specified threshold and granularity, based on the digital culture domain knowledge graph, the concept entity set is expanded to obtain an expanded entity set, and the digital culture domain knowledge graph is updated based on the expanded entity set.

[0107] Specifically, continuing the example in S104, the expanded concept entity set {Dream of Red Mansions, Cao Xueqin, Ming and Qing Novels} can be further expanded according to the pre-set threshold and granularity. The threshold here generally refers to the standard value for judging whether it is similar during the similarity comparison process. If it is greater than the threshold, it is considered similar, and if it is less than the threshold, it is considered dissimilar. For example, if the threshold is set to 60%, it is considered that the similarity is greater than 60%, and it is determined to be similar; if the similarity is less than 60%, it is determined to be dissimilar.

[0108] The purpose of setting granularity is to prevent concepts from being expanded indefinitely, but to have certain boundaries, which are not fixed. For example, sometimes users need to expand a large amount of data, and the required concept entities may be relatively more, which requires the granularity to be set larger; on the contrary, if users only need a brief summary, the granularity can be set smaller.

[0109] For the use of granularity, see Figure 4 , Figure 4 The granularity selection process according to an embodiment is shown. When the granularity is set to 1 (i.e., entities that are one step away from "Dream of Red Mansions"), the selected expanded entity set may include {Dream of Red Mansions, Cao Xueqin, Lin Daiyu, Tianqiao Theater, National Ballet of China}, which corresponds to all conceptual entities in the inner ring 41; when the granularity is set to 2 (i.e., entities that are two steps away from "Dream of Red Mansions"), the selected expanded entity set may include {Dream of Red Mansions, Cao Xueqin, Lin Daiyu, Tianqiao Theater, National Ballet of China, Swan Lake}, which corresponds to all conceptual entities in the outer ring 42. It can be seen that when the granularity is set to 2, the conceptual entities will be more than when the granularity is set to 1, which will further affect the number of summaries that can be generated later.

[0110] In one embodiment, according to a specified threshold and granularity, based on the digital culture domain knowledge graph, the concept entity set is expanded to obtain an expanded entity set, and the digital culture domain knowledge graph is updated based on the expanded entity set, including:

[0111] While forming the concept entity set, if the similarity between the concept entity and the entity in the digital culture field knowledge graph meets the specified threshold and granularity, the concept entity is incorporated into the concept entity set to obtain the expanded entity set.

[0112] Specifically, similar entities in the digital culture domain knowledge graph are expanded into the concept entity set, which can actually be done while the concept entity set is being formed. That is to say, in the digital culture domain knowledge graph, when searching for concept entities associated with each tag in the first cross-modal tag set, once it is found that the similarity requirements are met and the specified threshold and granularity requirements are met, the entity in the digital culture domain knowledge graph is directly expanded into the expanded entity set, which can greatly improve the efficiency of concept expansion.

[0113] For the newly added entities in the expanded entity set, if there is no corresponding concept in the digital culture domain knowledge graph, the entity can be expanded into the digital culture domain knowledge graph. At the same time, the relationship between these newly expanded entities and the original entities in the digital culture domain knowledge graph can be realized with the help of knowledge reasoning. For the implementation of knowledge reasoning, please refer to the introduction of the second half of step S107.

[0114] S106: Obtain a second cross-modal label set based on the vertical domain multimodal data, compare the expanded entity set with the second cross-modal label set, and sort the expanded entity set to obtain a sorted entity set.

[0115] In one embodiment, a second cross-modal label set is obtained based on the vertical domain multimodal data, the expanded entity set is compared with the second cross-modal label set, and the expanded entity set is sorted to obtain a sorted entity set, including:

[0116] Sampling the data perpendicular to the multimodal data to obtain a second cross-modal label set;

[0117] For each entity in the expanded entity set, calculating a weight of the entity in the second cross-modal tag set;

[0118] The expanded entity set is sorted according to the frequency of occurrence of the entity in the second cross-modal tag set to obtain a sorted entity set.

[0119] Figure 3 FIG. 2 shows a data flow diagram according to an embodiment of the present application. Figure 3 As shown, in the digital culture domain knowledge graph 5, the concept entities associated with each tag in the second cross-modal tag set 6 are searched to form a concept entity set 71, and the concept entity set 71 is expanded to obtain an expanded entity set 72. The expanded entity set 72 is compared with the second cross-modal tag set 6, and the expanded entity set 72 is sorted to obtain a sorted entity set 73.

[0120] The significance of comparing the expanded entity set 72 with the second cross-modal label set 6 is that if only conventional means are used to expand the concept entity set with the help of the digital culture field knowledge graph, some important information in the original second cross-modal label set 6 may be missed, which will affect the accuracy of the summary expansion and cause concept deviation.

[0121] Continuing with the example of "Dream of Red Mansions", in the process from the vertical domain multimodal data for generating the summary to the concept entity set, the concept of "Gao E" may be lost. At this time, if the expanded entity set 72 is compared with the second cross-modal label set 6, the concept of "Gao E" can be re-expanded into the expanded entity set 72 to avoid the concept of "Gao E" being missed.

[0122] Specifically, the essence of comparison is actually a process of comparing the augmented entity set with the second cross-modal label set, which can be achieved through the following process:

[0123] First, according to different modalities, random label extraction is performed on the vertical domain multimodal data to obtain a second cross-modal label set. For example, for text-type data, paragraphs or sentences in the text file can be randomly extracted, or the entire text file can be segmented to obtain a second cross-modal label set; for image-type data, several areas in the image can be randomly extracted, or image recognition can be performed on the entire image to obtain a second cross-modal label set; for audio-type data, several paragraphs in the audio file can be randomly captured, transcribed into text, and the second cross-modal label set can be obtained after segmentation; and for video-type data, the second cross-modal label set can also be obtained by randomly capturing video frames and then performing image recognition.

[0124] Second, determine whether each tag in the second cross-modal tag set is similar to the entity in the expanded entity set. Traverse the expanded entity set to determine whether each tag in the second cross-modal tag set is in the expanded entity set. This judgment process does not require completeness, but can be determined by semantic comparison to determine whether the maximum value of the semantic similarity between each tag and each entity in the expanded entity set reaches a specified threshold. If it exceeds the specified threshold, it is determined to be similar; if it is less than the specified threshold, it is determined to be dissimilar.

[0125] Third, expand the expanded entity set. For tags that are judged to be similar, it is considered that the concept corresponding to this tag is actually already in the expanded entity set. Even if it is forced to be added, it will increase the computational complexity when the content is expanded later. Therefore, for similar tags, choose not to operate. For dissimilar tags, the minimum tolerance set by the user can be used to determine whether this tag is useless for the subsequent summary expansion. In other words, when the maximum similarity between a tag and each entity in the expanded entity set is less than the specified threshold and greater than the minimum tolerance, the tag is expanded into the expanded entity set.

[0126] Fourth, sort according to the comparison results. In the process of expanding the entity set in the third step, although no operation is performed on similar tags, this does not mean that the entity is not important. Instead, it means that the concept is often mentioned and should be highlighted in the process of generating summaries. For such entities, you can choose to increase their weights so that they are at the front of the final sorted entity set. In this way, when generating summaries, these entities with higher weights and higher rankings will be given higher priority, whether in terms of the breadth of expansion or the position in the summary. It is worth noting that in order to improve efficiency, the sorting process can be carried out simultaneously with the process of expanding the entity set in the third step. That is to say, after finding similar entities in the expanded entity set, their weights can be increased at the same time and placed in a position that matches their weight.

[0127] For example, the augmented entity set currently includes {Lin Daiyu, Dream of the Red Chamber, Cao Xueqin}. After random label extraction of vertical domain multimodal data, the second cross-modal label set {The Story of the Stone, Dream of the Red Chamber, Grand View Garden} is obtained. It can be seen that "The Story of the Stone" and "Grand View Garden" are obviously "forgotten" labels after entering the vertical domain model. However, after comparing the second cross-modal label set with the augmented entity set, it is found that the similarity between "The Story of the Stone" and "The Dream of the Red Chamber" is greater than the specified threshold, so "The Story of the Stone" does not need to be included in the augmented entity set again. However, for "Grand View Garden", although its similarity with each entity in {Lin Daiyu, Dream of the Red Chamber, Cao Xueqin} is not as high as that of "The Story of the Stone", it is still greater than the minimum tolerance. Therefore, "Grand View Garden" is also included in the augmented entity set, and finally the augmented entity set is expanded to {Lin Daiyu, Dream of the Red Chamber, Cao Xueqin, Grand View Garden}. At the same time, since "The Story of the Stone" and "A Dream of Red Mansions" are similar, the weight of "A Dream of Red Mansions" will be increased, and the expanded entity set {Lin Daiyu, A Dream of Red Mansions, Cao Xueqin, Grand View Garden} will be sorted to obtain the sorted entity set {A Dream of Red Mansions, Lin Daiyu, Cao Xueqin, Grand View Garden}.

[0128] S107: Generate summary text based on the sorted entity set and the updated digital culture field knowledge graph.

[0129] In one embodiment, generating summary text based on the sorted entity set and the updated digital culture domain knowledge graph includes:

[0130] According to the ranking in the sorted entity set, each entity in the sorted entity set is expanded according to the specified summary expansion granularity to obtain an entity expansion sequence;

[0131] For each entity in the entity expansion sequence, a corresponding relationship is extracted in the digital culture field knowledge graph, and a summary text is generated in combination with the entity expansion sequence.

[0132] Specifically, the text generation technology commonly used in artificial intelligence (such as Transformer and pre-trained language models) can meet the process of summary generation. It is worth noting that in the process of summary generation, external knowledge can also be referenced. In this case, specific knowledge encoders need to be set for this external knowledge. The choice of knowledge encoder can be based on the data structure of external knowledge. When introducing pictures and videos as external knowledge, graph neural networks, convolutional neural networks, pre-trained language models, etc. can be used accordingly.

[0133] It is worth noting that entities, attributes and relationships are used when generating summary texts. In fact, entities and attributes are already quite sufficient through the expansion of this technical solution. For relationships, especially entities added from the second quadramodal label, the relationship between these entities and the original entities can be realized with the help of knowledge reasoning methods. For example, logic-based reasoning, graph-based reasoning, machine learning-based reasoning, etc. can all be used to realize knowledge reasoning. At present, logic-based knowledge methods, such as first-order predicate logic, description logic, etc.; graph-based methods, such as Path Ranking Algorithm (PRA, path sorting algorithm) and Association Rule Mining under Incomplete Evidence (AMIE, association rule mining of incomplete knowledge base); machine learning-based reasoning, such as using TransE, TransH and other models, have achieved relatively ideal results in entity relationship reasoning.

[0134] In one embodiment, the method further comprises:

[0135] The summary text is machine translated to obtain a summary translation text in a specified language.

[0136] Specifically, the summary text can be translated into a version in a specified language, such as English, Japanese, French, etc., through the currently very mature machine translation technology, so as to meet the more international needs of different users.

[0137] like Figure 3 As shown, the summary text 74 is the final generated result. It can be seen that the dotted box 7 is actually the core of this technical solution. The key point of this technical solution is to use the knowledge graph in the digital culture field, cooperate with the comparison of the expanded entity set and the second cross-modal label set, to improve the accuracy of summary generation.

[0138] A second aspect of the present invention also relates to a digital culture field query method, characterized in that it includes:

[0139] Presenting a query entry on a human-computer interaction interface;

[0140] In response to a user inputting a multimodal data query request instruction at the query entry, obtaining a query result;

[0141] In response to a user clicking on the query result, according to any one of the methods according to the first aspect of the claim, obtaining a summary corresponding to the query result;

[0142] The summary is presented on the human-computer interaction interface in the form of thumbnails.

[0143] The third aspect of the present invention also relates to an electronic device, comprising at least one processor and a memory, wherein the memory stores computer-executable instructions, and the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the digital culture field abstract generation method described in any one of the first aspect, or the query method described in the second aspect.

[0144] The fourth aspect of the present invention also relates to a computer-readable storage medium, in which multiple program codes are stored, characterized in that the program code is suitable for being loaded and run by a processor to execute the digital culture field abstract generation method described in any one of the first aspects, or the query method described in the second aspect.

[0145] The functions of each module in each system of the embodiment of the present invention can refer to the corresponding description in the above method, which will not be repeated here.

[0146] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0147] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.

[0148] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0149] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0150] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0151] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0152] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of various changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating abstracts in the field of digital culture, characterized in that: include: Obtaining vertical domain multimodal data for which summaries are to be generated; Inputting the multimodal data into a vertical domain model to generate a first cross-modal label set corresponding to each of the multimodal data, wherein the vertical domain model is obtained based on a base large model; Associating the first cross-modal tag set to a digital culture domain knowledge graph; Based on the digital culture field knowledge graph, searching for concept entities associated with the first cross-modal tag set to form a concept entity set; According to the specified threshold and granularity, based on the digital culture domain knowledge graph, the concept entity set is expanded to obtain an expanded entity set, and the digital culture domain knowledge graph is updated based on the expanded entity set; Obtaining a second cross-modal label set based on the vertical domain multimodal data, comparing the expanded entity set with the second cross-modal label set, and sorting the expanded entity set to obtain a sorted entity set; Based on the sorted entity set and the updated digital culture domain knowledge graph, a summary text is generated.

2. The method according to claim 1, characterized in that The step of obtaining the vertical domain model based on the large base model includes: The base large model is transformed to form a target large model, wherein the target large model sets a text encoder, an image encoder, an audio encoder and a video encoder at the original input layer of the base large model to extract features of samples of corresponding types; an input projector is set to project the extracted features into a common feature space; feature fusion is realized by using the interaction between the self-attention mechanism of the base large model and the feedforward network; a classifier is set after the original output layer of the base large model to generate cross-modal labels; a knowledge extractor is set after the classifier to extract entities and relationships from the labels; Pre-training the target large model to obtain a pre-trained target large model; Fine-tune the pre-trained target large model to obtain the vertical domain model.

3. The method according to claim 1, characterized in that Based on the digital culture domain knowledge graph, conceptual entities associated with the first cross-modal tag set are searched to form a conceptual entity set: Performing conceptual entity recognition on the tags associated with the first cross-modal tag set by a knowledge extractor to obtain the conceptual entity; By comparing similarities, determining whether the conceptual entity is similar to an entity in the knowledge graph in the digital culture field; In the case where the conceptual entity is similar to an entity in the digital culture field knowledge graph, the conceptual entity is included in the conceptual entity set.

4. The method according to claim 1, characterized in that: According to the specified threshold and granularity, based on the digital culture domain knowledge graph, the concept entity set is expanded to obtain an expanded entity set, and the digital culture domain knowledge graph is updated based on the expanded entity set, including: While forming the concept entity set, if the similarity between the concept entity and the entity in the digital culture field knowledge graph meets the specified threshold and granularity, the concept entity is incorporated into the concept entity set to obtain the expanded entity set.

5. The method according to claim 1, characterized in that A second cross-modal label set is obtained based on the vertical domain multimodal data, the expanded entity set is compared with the second cross-modal label set, and the expanded entity set is sorted to obtain a sorted entity set, including: Sampling the data perpendicular to the multimodal data to obtain a second cross-modal label set; For each entity in the expanded entity set, calculating a weight of the entity in the second cross-modal tag set; The expanded entity set is sorted according to the frequency of occurrence of the entity in the second cross-modal tag set to obtain a sorted entity set.

6. The method according to claim 1, characterized in that Based on the sorted entity set and the updated digital culture domain knowledge graph, a summary text is generated, including: According to the ranking in the sorted entity set, each entity in the sorted entity set is expanded according to the specified summary expansion granularity to obtain an entity expansion sequence; For each entity in the entity expansion sequence, a corresponding relationship is extracted in the digital culture field knowledge graph, and a summary text is generated in combination with the entity expansion sequence.

7. The method according to claim 1, characterized in that The method further comprises: The summary text is machine translated to obtain a summary translation text in a specified language.

8. A digital cultural field query method, characterized in that: include: Presenting a query entry on a human-computer interaction interface; In response to a user inputting a multimodal data query request instruction at the query entry, obtaining a query result; In response to a user clicking on the query result, obtaining a summary corresponding to the query result according to the method according to any one of claims 1 to 7; The summary is presented on the human-computer interaction interface in the form of thumbnails.

9. An electronic device, characterized in that: It includes at least one processor and a memory, wherein the memory stores computer-executable instructions, and the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the digital culture field abstract generation method described in any one of claims 1 to 7, or the query method described in claim 8.

10. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the digital culture field abstract generation method according to any one of claims 1 to 7, or the query method according to claim 8.

Images