Large model-based content automatic provenance generation system and method

Abstract

The application discloses a content automatic tracing generation system and method based on a large model, and belongs to the field of artificial intelligence.The technical scheme is as follows:the system comprises an information identification module, which is used for acquiring input information of a user and identifying the type of the input information, the type including text, pictures and / or audio; a search module, which is used for searching related information of the input information in a preset database; a fusion module, which comprises a knowledge fusion module, the knowledge fusion module being used for fusing a feature vector of the input information with a feature vector of the related information to obtain a fused feature vector; and a tracing large model, which is used for obtaining a tracing result of the input information according to the fused feature vector.The application improves the accuracy of the tracing result by fusing the input information with data in the preset database, and can simultaneously output various types of tracing results by designing branches of the tracing large model, thereby meeting the diversified needs of users.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, more particularly to a large model-based content automatic traceability generation system and method. BACKGROUND

[0002] With the popularity of the Internet and the rapid development of information technology, the amount of information on the network is growing exponentially. Social media, self-media platforms, various websites, etc. will generate a large amount of text, image and other content every day. People enjoy the convenience of information access, but also face the huge challenge of filtering and identifying true and false information. And today's information is no longer limited to a single text form, image, audio and other multi-modal content have gradually become mainstream, and their traceability needs in the process of dissemination and use have become increasingly prominent.

[0003] And the existing traceability method usually only involves traceability of one type of data, for example, the Chinese patent with authorization announcement number CN114138784B, which provides a storage-based information traceability method, device, electronic equipment and medium, which traces information through the search path of the source file, but the invention only involves the traceability of text content, so the existing technology has deficiencies. SUMMARY

[0004] In view of the deficiencies of the prior art, the purpose of the present application is to provide a large model-based content automatic traceability generation system and method, which performs multi-modal fusion on multiple types of data through a feature fusion unit, and fuses the input information with the information in the preset database through a knowledge fusion unit, enriches the traceability clues, improves the accuracy of traceability, and through the branch design of the traceability large model, multiple types of traceability results can be output at the same time, meeting the diversified needs of users.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] The present application provides a large model-based content automatic traceability generation system, comprising:

[0007] An information recognition module is configured to obtain input information of a user and recognize the type of the input information, wherein the type includes text, picture and / or audio;

[0008] A search module is configured to search for related information of the input information in a preset database;

[0009] A fusion module, the fusion module comprises a knowledge fusion unit, the knowledge fusion unit is used for fusing the feature vector of the input information with the feature vector of the related information, and obtaining a fusion feature vector;

[0010] The traceability large model is used to obtain a traceability result of the input information according to the fusion feature vector.

[0011] As a further improvement of the present application, the traceability result includes original text, source identification, time identification and authenticity identification, and the authenticity identification is used to represent the authenticity of the input information.

[0012] As a further improvement of the present application, the output layer of the traceability large model adopts an independent output branch design, each branch shares bottom layer parameters, each branch corresponds to an output task, and each output task corresponds to a result type.

[0013] As a further improvement of the present application, the preset database at least includes one of a vector database, a relational database and a knowledge graph database.

[0014] As a further improvement of the present application, the content automatic traceability generation system based on a large model further includes a feature extraction module, which is used to perform feature extraction on the input information according to the type of the input information, and output a feature vector corresponding to the type, and the feature extraction module includes a text feature extraction unit, a picture feature extraction unit and an audio feature extraction unit.

[0015] As a further improvement of the present application, the fusion module further includes a feature fusion unit, which is used to perform multi-modal fusion on the feature vectors output by the feature extraction module when the type of the input information is multiple, to obtain a feature vector of the input information, and the feature fusion unit includes an inter-modal attention subunit, an intra-modal attention subunit and a fusion subunit, the inter-modal attention subunit is used to assign weights to feature vectors of different types, the intra-modal attention subunit is used to assign weights to elements in the feature vectors of different types, and the fusion subunit is used to fuse the feature vectors of different types processed by the inter-modal attention subunit and the intra-modal attention subunit, to obtain the feature vector of the input information.

[0016] As a further improvement of the present application, the content automatic traceability generation system based on a large model further includes a verification module, which is used to measure the accuracy of the traceability result according to the input information and the original text.

[0017] The present application provides a content automatic traceability generation method based on a large model, which comprises:

[0018] Obtaining input information of a user and identifying the type of the input information, the type including text, picture and / or audio;

[0019] Searching for related information of the input information in a preset database;

[0020] According to the input information and the related information, a fusion feature vector is obtained;

[0021] According to the fusion feature vector, a traceability result of the input information is obtained, and the traceability result includes original text, source identification, time identification and authenticity identification;

[0022] According to the traceability result, the input information and a preset similarity function, the accuracy of the traceability result is judged.

[0023] As a further improvement of the application, if the type of the input information is multiple, the fusion feature vector is obtained according to the input information and the related information, including:

[0024] Different types of feature vectors are obtained according to the input information;

[0025] The different types of feature vectors are fused to obtain the feature vector of the input information;

[0026] The feature vector of the input information and the feature vector of the related information are fused to obtain the fusion feature vector.

[0027] As a further improvement of the application, if the type of the input information is text, the similarity function is determined according to the TF-IDF value, weight factor and cosine similarity of each feature word in the input information.

[0028] The application fuses multi-aspect data through the knowledge fusion module and the feature fusion module, makes up for the deficiency of single feature, narrows the traceability range, improves the traceability accuracy and efficiency, and through the branch design of the traceability large model, can output multiple types of traceability results at the same time, which is used for constructing a very comprehensive "image" about the input information, and is convenient for traceability analysis from multiple angles. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 It is a system structure schematic diagram of the application;

[0030] Figure 2 It is a structure schematic diagram of the feature extraction module in the application;

[0031] Figure 3 It is a structure schematic diagram of the fusion module in the application;

[0032] Figure 4 It is a structure schematic diagram of the feature fusion unit in the application;

[0033] Figure 5 It is a method step schematic diagram of the application.

[0034] Label: 110, information identification module; 120, search module; 130, fusion module; 140, traceability large model; 150, verification module; 131, feature fusion unit; 132, knowledge fusion unit; 1311, inter-module attention subunit; 1312, intra-module attention subunit; 1313, fusion subunit; 160, feature extraction module; 161, text feature extraction unit; 162, picture feature extraction unit; 163, audio feature extraction unit. DETAILED DESCRIPTION

[0035] The technical solutions of the present application will be described in detail below with the help of the drawings and specific embodiments. It should be understood that the embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solutions of the present application, but not limitations of the technical solutions of the present application.

[0036] The term "and / or" in the following merely describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " generally represents that the associated objects before and after it are in an "or" relationship.

[0037] As shown in Figure 1 The embodiment of the present application provides a content automatic traceability generation system based on a large model, which comprises:

[0038] The information identification module 110 is configured to obtain input information of a user and identify the type of the input information, the type including text, picture and / or audio;

[0039] The search module 120 is configured to search for related information of the input information in a preset database;

[0040] The fusion module 130 comprises a knowledge fusion unit 132, which is configured to fuse the feature vector of the input information with the feature vector of the related information to obtain a fusion feature vector;

[0041] The traceability large model 140 is configured to obtain a traceability result of the input information according to the fusion feature vector.

[0042] The system provided by the embodiment integrates multi-source information by fusing the input information with the data in the preset database, excavates the potential association between the input information and the information stored in the database, makes up for the limitations of relying only on the input information itself for traceability, and improves the accuracy of traceability.

[0043] Further, the preset database at least comprises one of a vector database, a relational database and a knowledge graph database.

[0044] The vector database is a collection of high-dimensional vectors representing entities or concepts, which are usually transformed from words, phrases, or documents, and each dimension of the vector represents a different feature or attribute. The relational database organizes data in the form of tables, each row of the table represents a record, and each column represents an attribute or field. The database establishes the relationship between data by defining the relationship between tables, and the data is usually structured data such as numerical values, strings, dates, etc. The database can be managed, retrieved, and operated through query languages such as SQL. The knowledge graph database is a collection of nodes and edges representing entities and their relationships. The nodes represent entities, and the edges represent the relationships between entities. The knowledge graph database can also contain attribute information of entities and relationships. The data in the knowledge graph has semantic, structured, and scalable features, and can represent knowledge in various fields, such as the relationships between people, places, events, organizations, etc.

[0045] Specifically, when the preset database is a vector database, the input information can be converted into a vector form, the semantic similarity between the input information and the database stored information can be measured according to the similarity between the vectors, and the relevant vector information retrieved from the database can be spliced with the vector form of the input information to obtain a fusion feature vector. For example, in a content management system of a news media, a large number of news articles are usually converted into vectors and stored in a vector database. When a new article needs to be traced, the system converts the article into a vector and searches for similar articles in the vector database. Assuming that the new article is about a flood disaster in a certain region, through the similarity search of the vector database, multiple articles about infrastructure construction or flood prevention preparation in the region can be found in the database. It can be inferred that the description of the impact of the disaster on the infrastructure in the new article may be related to these previous reports, which is a continuation and feedback of the previous situation after the disaster. Then, the feature vectors can be extracted from the texts related to infrastructure construction or flood prevention preparation and spliced with the vector of the new article, which helps to trace the background and basis of the content of the new article from the time dimension.

[0046] When the preset database is a relational database, the input information can be extracted and stored in a structured form in a relational table, facilitating rule-based queries, statistics, and analysis. For example, in a commodity comment management system of an e-commerce platform, the comment content can be stored in different relational tables with user information and commodity information, respectively. Through the association query of the relational database, all comments of a specific commodity or the comment history of a specific user can be obtained. Assuming that the source of a malicious comment needs to be traced, the possible comment records can be filtered according to the keywords in the comment, and then the user's registration information and purchase records can be obtained by associating the user table. Then, the information feature vectors of multiple users are formed according to the comment content, the user's registration information, and the purchase records. Through the search of the tracing large model 140 on each feature vector, the publisher of the comment can be accurately tracked.

[0047] When the preset database is a knowledge graph, based on the relationships and reasoning capabilities in the knowledge graph, the meaning of the input information can be better understood and the tracing task can be performed. For example, assuming that the input information is about the production background of a certain movie, the movie (as an entity), its director (a related entity), and the type it belongs to (a relationship) can be found according to the knowledge graph. Then, the entities and relationships of the knowledge graph are encoded into vectors through embedding technology, and are spliced with the feature vectors obtained from the original input information to obtain a fusion feature vector, which is used for subsequent tracing tasks.

[0048] Further, the tracing result includes the original text, the source identifier, the time identifier, and the authenticity identifier.

[0049] Specifically, the source identifier shows the website or social media platform from which the input text comes. The time identifier shows the publication time of the input information. The authenticity identifier is used to represent the authenticity of the input information, for example, when the input image is PS processed, the input information is identified as false information, or when the input text is an AI synthesized text, the input information is identified as false information.

[0050] Further, the output layer of the tracing large model 140 adopts an independent output branch design, each branch shares bottom layer parameters, each branch corresponds to an output task, and each output task corresponds to a result type, for example, one branch of the output layer is used to output the source identifier, and the other branch is used to output the authenticity identifier.

[0051] Specifically, the tracing large model 140 generally includes a network layer and an output layer. The independent output branch design adopted in this embodiment shares parameters in the network layer, reduces the difference interference between tasks, and then performs detailed task processing according to the characteristics of different tasks in each branch.

[0052] When the traceability large model 140 simultaneously performs multiple output tasks, due to the different tasks having their own unique objectives, evaluation indicators, and expected output forms, for example, the source identifier can contain specific platform names or creator information, the authenticity identifier can contain yes or no and the corresponding confidence, etc., the existence of these differences makes it more difficult for all output tasks to coexist reasonably under one model framework, and the search direction of different tasks may not be consistent, which can easily result in the output results of a certain task being accurate while the output results of other tasks being inaccurate.

[0053] The method provided by the embodiment reduces the conflicts caused by task differences, focuses on general data features that are helpful to all tasks at the network layer, and avoids mutual interference caused by too large task target differences from the beginning. For example, the task for outputting the source identifier focuses on mining the relationship between the content and the external entity, while the task for outputting the authenticity identifier focuses on whether the content is tampered with. If each task independently processes the original data, it is easy to lose one and gain the other. By sharing the bottom layer parameters, each task searches based on a common information base, which to some extent alleviates the conflicts caused by such differences and enables each task to develop more orderly towards its own target.

[0054] On the other hand, the progress of each task can be balanced, so that each task can complete the output task in the branch of the output layer according to the difficulty of its own task, data characteristics and other factors based on the shared bottom layer parameters, which is more conducive to balancing the progress of each task. For example, the task for outputting the time identifier can quickly complete the preliminary search based on the existing features, which takes less time and is easy to affect the output speed of other incomplete tasks, while the task for outputting the authenticity identifier needs to consider more associated information, which takes more time and is easy to be affected. At this time, the task for outputting the authenticity identifier can further mine the associated information by using the multi-layer network structure in the branch, so that each task can output the optimal result.

[0055] Further, as shown in Figure 2 The content automatic traceability generation system based on the large model further includes a feature extraction module 160 for performing feature extraction on the input information according to the type of the input information and outputting a feature vector corresponding to the type. The feature extraction module 160 includes a text feature extraction unit 161, a picture feature extraction unit 162, and an audio feature extraction unit 163.

[0056] Specifically, the extraction of the text feature vector can adopt a bag-of-words model, a TF-IDF (Term Frequency-Inverse Document Frequency) method, etc. The basic principle of the bag-of-words model is to regard a text as a set of words, ignore the order and grammatical structure of the words, and only focus on the frequency of the occurrence of the words. For example, assuming that a dictionary contains five words: "apple", "banana", "orange", "eat", and "buy", and a text content is "I bought apples and bananas", the feature vector obtained by the bag-of-words model is [1, 1, 0, 0, 1], which respectively corresponds to the frequency of the occurrence of the above words in the text. The TF-IDF takes into account the importance weight of the words on the basis of the bag-of-words model. For example, assuming that in a document set, "apple" appears in 8 of 10 documents, "banana" appears in 3 of 10 documents, "orange" appears in 5 of 10 documents, "eat" appears in 4 of 10 documents, and "buy" appears in 6 of 10 documents, the TF value of "apple" for the text "I bought apples and bananas" is 1 / 3 (assuming that the total number of words in the text is 3), and the IDF value of "apple" is log(10 / 8) (taking the natural logarithm as an example), so the TF-IDF value of "apple" is the product of the two, and the TF-IDF values of the other words are calculated in the same way to form the TF-IDF feature vector of the text.

[0057] The extraction of the picture feature vector can be based on a color histogram of the picture or a local binary pattern of the picture. The basic principle of the method based on the color histogram of the picture is to count the frequency of the occurrence of different colors in the image and take it as a kind of feature representation of the picture. For example, the color space (such as RGB, HSV, etc.) is divided into several intervals, and then the proportion of the number of pixel points in each interval to the total number of pixel points is calculated to form a feature vector. The principle of the method based on the local binary pattern is to compare the neighborhood pixels with each pixel as the center, generate a binary code according to the comparison result, and then count the frequency of the occurrence of different coding modes to construct a feature vector.

[0058] The audio feature vector can be obtained by the frequency domain feature of the audio. For example, the discrete Fourier transform can be performed on the audio signal to convert the time domain audio signal into a frequency domain representation, obtain the frequency spectrum of the audio signal, and intercept part of the frequency amplitude in the frequency spectrum as the feature vector. Or the audio frequency spectrum is analyzed on the mel frequency scale, the linear frequency is converted into the mel frequency, the logarithm is taken and the discrete cosine transform (DCT) is performed to obtain the mel frequency cepstrum coefficient, and the first 10-20 coefficients are taken to form the feature vector.

[0059] Further, as Figure 3As shown, the fusion module 130 further includes a feature fusion unit 131, which is configured to perform multi-modal fusion on the feature vectors output by the feature extraction module 160 when the type of the input information is multiple, to obtain a feature vector of the input information.

[0060] Specifically, as shown in FIG. 1, the feature fusion unit 131 includes an inter-modal attention subunit 1311, an intra-modal attention subunit 1312, and a fusion subunit 1313. Figure 4 The inter-modal attention subunit 1311 is configured to assign weights to the feature vectors of different types, the intra-modal attention subunit 1312 is configured to assign weights to the elements within the feature vectors of different types, and the fusion subunit 1313 is configured to fuse the feature vectors of different types processed by the inter-modal attention subunit 1311 and the intra-modal attention subunit 1312, to obtain a feature vector of the input information.

[0061] Specifically, the inter-modal attention subunit 1311 assigns weights to the feature vectors of different types by first mapping the feature vectors of different modalities, i.e., different types, to a unified dimensional space through a fully connected layer or linear transformation. For example, the text feature vector and the image feature vector may be mapped to -dimensional spaces, respectively:

[0062] ;

[0063] wherein and represent real number spaces of and dimensions, respectively, and are the result vectors of the text feature vector and the image feature vector after linear transformation and mapping to the -dimensional space, and are learnable parameter matrices for the text feature vector and the image feature vector, respectively, with dimensions of and , and are bias terms in the linear transformation of the text feature vector and the image feature vector, respectively.

[0064] Then, a cross-attention mechanism is used, with one modality as the query (Query) and the other modality as the key (Key) and value (Value). For example, taking text as the query and image as the key and value:

[0065] ;

[0066] The attention score is calculated by dot product, and the weighted fusion result of the text to the image feature is output:

[0067] ;

[0068] wherein, is a short form of Query, is a short form of Key, is a short form of Value, represents a calculation function of cross-attention mechanism, which takes Query, Key and Value as input, and outputs a weighted fusion result, represents an activation function, which is used to normalize the result after dot product operation.

[0069] Finally, the cross-modal attention result is spliced or weighted summed with the original feature to obtain the cross-modal fusion feature :

[0070] ;

[0071] wherein, represents a learnable parameter.

[0072] The step of the intra-modal attention subunit 1312 for assigning weights to the elements in the feature vectors of different types is that, firstly, self-attention calculation is performed on the feature vectors of the same modality, for example, self-attention calculation is performed on the text feature vector and the output formula of the re-weighted representation of the text internal feature is:

[0073] ;

[0074] ;

[0075] wherein, , and are learnable parameter matrices for transforming the text feature vector into , , .

[0076] Then, the original feature information is reserved through a residual connection to prevent information loss, and the specific formula is:

[0077] ;

[0078] wherein, represents the final output obtained after layer normalization of the result of the residual connection, represents a layer normalization operation, denotes a regularization operation.

[0079] The fusion subunit 1313 fuses the feature vectors of different types processed by the inter-modal attention subunit 1311 and the intra-modal attention subunit 1312 to obtain a feature vector of the input information The steps are as follows:

[0080] ;

[0081] wherein, is a learnable weight parameter generated through a fully connected layer.

[0082] Further, the above basic attention mechanism can be replaced by a multi-head attention mechanism, that is, the Query, Key and Value are split into multiple heads, so that the attention mechanism model can learn the attention distribution in different subspaces in parallel, thereby enhancing the model's ability to capture different features and information; the above text feature matrix can be combined with position encoding, that is, the original feature matrix and the position encoding matrix are added element by element to obtain a new feature matrix, and the above steps are performed based on the new feature matrix, so that the attention mechanism model can utilize the position information of the elements when processing the feature matrix, thereby better understanding the semantics of the text sequence.

[0083] For example, if the input is a news text about a flood (text modality) and a picture (image modality), the inter-modal attention subunit 1311 can highlight the association between the text describing the flood and the water area in the picture; the intra-modal attention subunit 1312 can focus the picture on the disaster area and the text on the part containing core words such as "flood", "disaster" and "rescue"; the fusion subunit 1313 can flexibly fuse the text features and image features processed by the inter-modal attention and intra-modal attention according to the specific circumstances of the text features and image features, using appropriate fusion methods (such as splicing, weighted summation, attention mechanism-based fusion, etc.), to obtain the final fusion feature vector.

[0084] The feature fusion unit 131 provided in this embodiment is used to fuse multiple types of features to enrich the trace clues and improve the accuracy of tracing. Meanwhile, the inter-modal attention subunit 1311 focuses on the cross-modal association information between different types of information, dynamically assigns weights to the features of different modalities, the intra-modal attention subunit 1312 excavates the feature information that is more significant in each type of information, enhances the expression of modal features, and finally the fusion subunit 1313 performs fusion. Compared with the simple feature splicing in the prior art, the method provided in this embodiment can fully fuse features of various types, take advantage of features of various types, significantly improve the precision of multi-modal fusion, and further improve the accuracy of tracing.

[0085] Further, the large model-based content automatic traceability generation system further comprises a verification module 150 configured to measure the accuracy of the traceability result according to the input information and the original text.

[0086] The large model-based content automatic traceability generation system provided by the embodiments of the present application enriches the traceability clues, improves the traceability accuracy, and simultaneously outputs multiple types of traceability results through the branch design of the traceability large model 140, so as to meet the diversified needs of users.

[0087] Further, as shown in Figure 5 The large model-based content automatic traceability generation method provided by the embodiments of the present application comprises the following steps:

[0088] Obtaining input information of a user and identifying the type of the input information, the type including text, picture and / or audio;

[0089] Searching for related information of the input information in a preset database;

[0090] Obtaining a fusion feature vector according to the input information and the related information;

[0091] Obtaining a traceability result of the input information according to the fusion feature vector, the traceability result including the original text, the source identifier, the time identifier and the authenticity identifier;

[0092] Measuring the accuracy of the traceability result according to the traceability result, the input information and a preset similarity function.

[0093] Further, the embodiments provide a step of obtaining a fusion feature vector according to input information and related information when the type of the input information is multiple, which comprises the following steps:

[0094] Obtaining feature vectors of different types according to the input information;

[0095] Performing multi-modal fusion on the feature vectors of different types to obtain a feature vector of the input information;

[0096] Fusing the feature vector of the input information with a feature vector of the related information to obtain the fusion feature vector.

[0097] The large model-based content automatic traceability generation method provided by the embodiments of the present application fuses the input information with the information in the preset database, improves the traceability accuracy and efficiency, and measures the accuracy of the traceability result through the similarity between the original text and the input information, so that the user can understand the reliability of the traceability result.

[0098] Specifically, if the type of the input information is text, the similarity function is:

[0099] ,

[0100] wherein, denotes the input information, denotes the original text, and denote the th and the th feature word in the input information, and denote the th and the th feature word in the original text, and are the TF-IDF values of and , and respectively, and denote the weight factors of and , the size of the weight factor is determined by the part of speech of the feature word, generally, the weight factor of the noun is the largest, and the weight factors of the verb and the adjective are smaller, and denote the cosine similarity of and , and and .

[0101] If the type of the input information is a picture, the similarity function is:

[0102] ;

[0103] wherein denotes the picture in the input information, denotes the original picture in the traceability result, and denote the pixel mean of and , and denote the pixel standard deviation of and , denotes the pixel covariance of and , wherein , are two constants, used to avoid the case that the denominator is zero.

[0104] If the type of the input information is audio, the comparison of the similarity between audios can adopt a Fourier transform-based manner, a waveform comparison-based manner, a Mel spectrum comparison-based manner, a Chroma comparison-based manner, etc. The Fourier transform-based manner converts a time-domain signal into a frequency-domain signal, and obtains the similarity degree by comparing the spectral features. The waveform comparison-based manner obtains the similarity degree between audios by calculating the similarity degree between shapes. The Mel spectrum comparison-based manner converts the audio into a Mel spectrum, and obtains the similarity degree of the audio by comparing the features of the Mel spectrum. The Chroma comparison-based manner converts the audio signal into a frequency-domain signal through Fourier transform, then performs Chroma analysis on the frequency-domain signal to obtain Chroma features, and measures the similarity between audios by comparing the Chroma features.

[0105] The content automatic tracing generation system and method based on a large model provided by the embodiments of the present application fuse data in multiple aspects through the knowledge fusion unit 132 and the feature fusion unit 131, make up for the deficiency of a single feature, narrow the tracing range, improve the accuracy and efficiency of tracing, and through the branch design of the tracing large model 140, multiple types of tracing results can be output at the same time, which is convenient for tracing analysis. In addition, the verification module 150 calculates the similarity between the input information and the tracing result to evaluate the accuracy of the tracing result.

[0106] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0107] The present application is described with reference to flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The functions specified in one flow or multiple flows and / or blocks

[0108] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow Figure 1 the function specified in the flow or flows and / or blocks Figure 1 the function specified in the flow or flows and / or blocks

[0109] The above description is merely that of preferred embodiments of the application, and the protection scope of the application is not limited to the above-mentioned embodiments. Any technical scheme falling within the concept of the application shall fall within the protection scope of the application. It should be noted that, for those of ordinary skill in the art, some improvements and refinements without departing from the principles of the application shall also be considered as falling within the protection scope of the application.

Claims

1. A large model-based content automatic provenance generation system, characterized in that, The information recognition module is configured to obtain input information of a user and identify a type of the input information, the type including text, a picture, and / or audio. The search module is configured to search for related information of the input information in a preset database. The feature extraction module is configured to perform feature extraction on the input information according to the type of the input information and output a feature vector corresponding to the type, the feature extraction module including a text feature extraction unit, a picture feature extraction unit, and an audio feature extraction unit. The fusion module includes a knowledge fusion unit and a feature fusion unit, the feature fusion unit being configured for a case where the type of the input information is multiple, the feature fusion unit including an inter-module attention subunit, an intra-module attention subunit, and a fusion subunit, the inter-module attention subunit being configured to map feature vectors of different types to a unified dimensional space and then assign weights to the feature vectors of different types according to a cross-attention mechanism, wherein the feature vector of the text type is obtained according to text features and position encoding, the intra-module attention subunit being configured to assign weights to elements within the feature vectors of different types according to self-attention calculation and residual connection, and the fusion subunit being configured to fuse the feature vectors of different types processed by the inter-module attention subunit and the intra-module attention subunit to obtain a feature vector of the input information; and the knowledge fusion unit being configured to fuse the feature vector of the input information and a feature vector of the related information to obtain a fusion feature vector. The traceability large model is configured to obtain a traceability result of the input information according to the fusion feature vector, an output layer of the traceability large model adopting an independent output branch design, each branch sharing bottom-layer parameters, each branch corresponding to an output task, and each output task corresponding to a result type, the result type including original text, a source identifier, a time identifier, and a true-false identifier, the true-false identifier being configured to represent authenticity of the input information. The preset database includes at least one of a vector database, a relational database, and a knowledge graph database.

2. The large model based content automatic provenance generation system of claim 1, wherein, The content automatic traceability generation system based on a large model further includes a verification module configured to measure accuracy of the traceability result according to the input information and the original text.

3. The large model based content automatic provenance generation system of claim 1, wherein, The information recognition module is configured to obtain input information of a user and identify a type of the input information, the type including text, a picture, and / or audio.

4. The method for automatically tracing the content based on a large model, applied to the system for automatically tracing the content based on a large model according to any one of claims 1-3, characterized in that, The search module is configured to search for related information of the input information in a preset database. The feature extraction module is configured to perform feature extraction on the input information according to the type of the input information and output a feature vector corresponding to the type, the feature extraction module including a text feature extraction unit, a picture feature extraction unit, and an audio feature extraction unit. The fusion module includes a knowledge fusion unit and a feature fusion unit, the feature fusion unit being configured for a case where the type of the input information is multiple, the feature fusion unit including an inter-module attention subunit, an intra-module attention subunit, and a fusion subunit, the inter-module attention subunit being configured to map feature vectors of different types to a unified dimensional space and then assign weights to the feature vectors of different types according to a cross-attention mechanism, wherein the feature vector of the text type is obtained according to text features and position encoding, the intra-module attention subunit being configured to assign weights to elements within the feature vectors of different types according to self-attention calculation and residual connection, and the fusion subunit being configured to fuse the feature vectors of different types processed by the inter-module attention subunit and the intra-module attention subunit to obtain a feature vector of the input information; and the knowledge fusion unit being configured to fuse the feature vector of the input information and a feature vector of the related information to obtain a fusion feature vector. The traceability large model is configured to obtain a traceability result of the input information according to the fusion feature vector, an output layer of the traceability large model adopting an independent output branch design, each branch sharing bottom-layer parameters, each branch corresponding to an output task, and each output task corresponding to a result type, the result type including original text, a source identifier, a time identifier, and a true-false identifier, the true-false identifier being configured to represent authenticity of the input information. The preset database includes at least one of a vector database, a relational database, and a knowledge graph database. The content automatic traceability generation system based on a large model further includes a verification module configured to measure accuracy of the traceability result according to the input information and the original text.

5. The large model-based content automatic provenance generation method of claim 4, wherein, ​ ​ The different types of feature vectors are fused in a multimodal manner to obtain a feature vector of the input information; The feature vector of the input information and the feature vector of the related information are fused to obtain the fused feature vector.

6. The large model-based content automatic provenance generation method of claim 4, wherein, If the type of the input information is text, the similarity function is determined according to a TF-IDF value, a weight factor and a cosine similarity of each feature word in the input information.

Images