Multi-modal large model driven character knowledge graph construction method and system

Abstract

The application discloses a multi-modal large model driven character knowledge graph construction method and system, relates to the field of knowledge graph modeling, and comprises the following steps: acquiring a target character name and multi-character attribute description information, and constructing a knowledge graph framework; acquiring corresponding target multi-source material data according to the target character name; for each character attribute description information, extracting at least one attribute matching material segment corresponding to the character attribute description information from the target multi-source material data, and performing fusion processing on each attribute matching material segment corresponding to the character attribute description information based on a multi-modal large model to output corresponding attribute multi-modal generation data; and filling each attribute multi-modal generation data into a corresponding attribute node in the knowledge graph framework according to the character attribute description information to construct a multi-modal character knowledge graph. Thus, a high-quality character knowledge graph generated by multi-modal data fusion is obtained through the driving of the multi-modal large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of knowledge graph modeling, and particularly relates to a multi-modal large model driven method and system for constructing a character knowledge graph. BACKGROUND

[0002] With the popularity of the Internet and mobile terminals, massive amounts of multi-modal data such as text, images, and videos are constantly being generated, providing rich resources for knowledge representation and management. As an important branch of knowledge graphs, character knowledge graphs not only have wide applications in social networks, public opinion analysis, digital humanities, and other fields, but also face the challenge of how to comprehensively and accurately depict the multi-dimensional characteristics of characters.

[0003] Traditional knowledge graph construction methods mainly rely on text information for entity extraction and relationship inference, but such single-modal data has problems such as insufficient information dimension, ambiguous semantics, and noise interference. Although text can express a large amount of semantic information, it often cannot fully describe the multi-dimensional characteristics of characters such as appearance, emotions, behavior, and environment. For example, the description of a character may only involve their profession, nationality, or part of their experience, while ignoring their visual image and behavioral characteristics. In addition, the noise such as spelling errors, implicit information, and colloquial expressions in text data will directly affect the accuracy of entity and relationship extraction, leading to ambiguous semantics or ambiguity in knowledge graphs constructed by pure text.

[0004] To address the above problems, the industry has not yet proposed a better technical solution. SUMMARY

[0005] The present application provides a multi-modal large model driven method and system for constructing a character knowledge graph, a storage medium, a computer program product, and an electronic device to at least solve the problem of information limitation and semantic ambiguity in constructing a knowledge graph by a single text in the related art.

[0006] In a first aspect, an embodiment of the present application provides a multi-modal large model driven character knowledge graph construction method, comprising: obtaining a target character name and multiple character attribute description information, and constructing a knowledge graph framework; the knowledge graph framework comprises a core node and multiple attribute nodes connected to the core node, the core node is used to indicate the target character name, and each attribute node is used to indicate corresponding character attribute description information; obtaining corresponding target multi-source material data according to the target character name, the target multi-source material data comprises text material, image material and video material matched with the target character name; for each character attribute description information, at least one attribute matching material segment corresponding to the character attribute description information is extracted from the target multi-source material data, and each attribute matching material segment corresponding to the character attribute description information is fused and processed based on a multi-modal large model to output corresponding attribute multi-modal generated data; the multi-modal generated data comprises text generated data, image generated data and video generated data; each attribute multi-modal generated data is filled into the corresponding attribute node in the knowledge graph framework according to the character attribute description information, so as to construct a multi-modal character knowledge graph.

[0007] In a second aspect, an embodiment of the present application provides a multi-modal large model driven character knowledge graph construction system, comprising: a graph framework construction unit, configured to obtain a target character name and multiple character attribute description information, and construct a knowledge graph framework; the knowledge graph framework comprises a core node and multiple attribute nodes connected to the core node, the core node is used to indicate the target character name, and each attribute node is used to indicate corresponding character attribute description information; a multi-source material obtaining unit, configured to obtain corresponding target multi-source material data according to the target character name, the target multi-source material data comprises text material, image material and video material matched with the target character name; an attribute data generating unit, configured to, for each character attribute description information, extract at least one attribute matching material segment corresponding to the character attribute description information from the target multi-source material data, and fuse and process each attribute matching material segment corresponding to the character attribute description information based on a multi-modal large model to output corresponding attribute multi-modal generated data; the multi-modal generated data comprises text generated data, image generated data and video generated data; a graph attribute filling unit, configured to fill each attribute multi-modal generated data into the corresponding attribute node in the knowledge graph framework according to the character attribute description information, so as to construct a multi-modal character knowledge graph.

[0008] In a third aspect, an electronic device is provided, which includes at least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the multi-modal large model driven character knowledge graph construction method of any of the embodiments of the present application.

[0009] In a fourth aspect, the embodiments of the present application provide a storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the multi-modal large model driven character knowledge graph construction method of any of the embodiments of the present application.

[0010] In a fifth aspect, the embodiments of the present application provide a computer program product, which includes computer programs / instructions, and the computer programs / instructions, when executed by a processor, implement the steps of the multi-modal large model driven character knowledge graph construction method of any of the embodiments of the present application.

[0011] Through the multi-modal large model driven character knowledge graph construction method and system provided by the present application, at least the following technical effects can be achieved:

[0012] (1) By introducing text, image, video and other multi-source material data, the multi-modal data is processed, the generated character attribute description can not only include structured information such as occupation and nationality, but also cover unstructured multi-modal information such as appearance characteristics, expression actions, and voice intonation, so that the character knowledge graph is more complete, three-dimensional and diverse in knowledge representation. In addition, using a multi-modal large model to perform semantic alignment and deep understanding on character attribute materials in different modalities, visual information can be used to assist verification of the accuracy of text content, effectively reducing entity recognition errors caused by spelling errors, colloquial expressions, semantic ambiguity, etc. in the text, and significantly improving the accuracy and stability of entity and relationship extraction in the knowledge graph.

[0013] (2) Using attribute nodes based on character attribute description information, combined with a multi-modal generation data filling mechanism, each attribute information can be independently and carefully displayed, thereby realizing multi-dimensional fine portrayal of character features and constructing a knowledge graph model framework with good structural hierarchy and extensibility, while supporting subsequent expansion and dynamic attribute management and updating of the character knowledge graph.

[0014] Through the technical solution, through the driving of the multi-modal large model, the high-quality character knowledge graph generated by the multi-modal data fusion is not only clear and rigorous in logical structure, but also more intuitive in display, significantly improving the comprehensiveness, accuracy and robustness of the knowledge graph construction, and providing strong support for deep mining and intelligent application of multi-modal character information. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0016] Figure 1 A flowchart of an example of a method for constructing a character knowledge graph driven by a multimodal large model according to an embodiment of the present application is shown;

[0017] Figure 2 Shown according to Figure 1 FIG. 1 is a schematic diagram of an exemplary operation flow of step S110 in FIG. 1 ;

[0018] Figure 3 Shown according to Figure 1 FIG. 1 is a schematic diagram of an exemplary operation flow of step S120 in FIG. 1 ;

[0019] Figure 4 A schematic diagram showing an operation flow of an example of extracting attribute-matched material segments from target multi-source material data according to an embodiment of the present application is shown;

[0020] Figure 5 A structural block diagram of an example of a multimodal large model-driven character knowledge graph construction system according to an embodiment of the present application is shown;

[0021] Figure 6 This is a schematic structural diagram of an embodiment of an electronic device of the present application. DETAILED DESCRIPTION

[0022] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] In the technical solutions of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved shall comply with the provisions of relevant laws and regulations and shall not violate public order and good morals.

[0024] Figure 1 A flowchart of an example of a method for constructing a character knowledge graph driven by a multimodal large model according to an embodiment of the present application is shown.

[0025] As to the execution subject of the embodiment method of the present application, it can be any controller or processor with computing or processing capability. Specifically, it can be implemented by a knowledge graph management platform or a knowledge graph management system, which generates a high-quality knowledge graph through multi-modal data fusion, especially through the construction of a knowledge graph attribute framework and the extraction and fusion processing of multi-modal attribute matching material segments for character attributes, realizes semantic enhancement of character attributes, and enables the knowledge graph to intuitively display multi-dimensional information of corresponding text, image and video for different character attribute nodes, meeting the needs of digital humanities, public opinion analysis and other fields for diversified data presentation.

[0026] In some examples, it can be integrated in an electronic device or terminal in the form of software, hardware or a combination of software and hardware, and the type of terminal or electronic device can be diversified, such as a mobile phone, a tablet computer or a desktop computer, etc.

[0027] As shown in FIG. 1, Figure 1 In step S110, the target character name and the multi-character attribute description information are obtained, and a knowledge graph framework is constructed.

[0028] Here, the knowledge graph framework includes a core node and a plurality of attribute nodes respectively connected to the core node, the core node is used to indicate the target character name, and each attribute node is used to indicate the corresponding character attribute description information.

[0029] In some embodiments, the name of the target character (such as "Zhang San") is explicitly determined through user input or system automatic recognition, and natural language processing (NLP) technology can be used for entity recognition (NER) on the input text to ensure the accuracy of the extracted target character name.

[0030] As to the character attribute description information, on the one hand, it can be a pre-set character feature type (such as occupation, hometown, hobby, etc.), so the standardized attribute structure can be agreed through system preset information. On the other hand, the character attribute description information can also be closely related to the target character, for example, some characters have individualized representative deeds (for example, obtaining a major scientific research achievement award, etc.), which can better realize the in-depth portrayal of the character, so external data sources (such as encyclopedias, news reports, etc.) can be used to expand the attribute description information related to the target character, and relationship extraction technology and semantic analysis technology can be used to convert unstructured text into structured character attribute description.

[0031] In terms of the construction details of the knowledge graph framework, it can be created by using a knowledge graph modeling tool (such as Neo4j), creating a core node for storing the target person's name, and creating an attribute node for each attribute description information, and then storing the relationship between the nodes and edges. Thus, by using the knowledge graph framework, the scattered attribute information of the person is organized in a structured form, a structured and semantically clear skeleton of the person knowledge graph is constructed, and unified expression and management of multi-dimensional attributes of the person are realized, providing clear target direction for subsequent data filling and semantic fusion. In addition, the knowledge graph framework supports the dynamic addition of new attribute nodes, can adapt to different task requirements, and improves the scalability and maintainability of the knowledge graph.

[0032] In step S120, the corresponding target multi-source material data is obtained according to the target person's name, and the target multi-source material data includes text material, image material and video material matching the target person's name.

[0033] In some embodiments, based on the target person's name, a search engine, a social media API, a content database, etc. are automatically called to retrieve and collect multi-source data related to the person, and the obtained data includes but is not limited to: news text, social dynamics, images (such as photos, portraits), video clips (such as interviews, speeches, documentaries). Further, a multi-modal data preprocessing module is introduced to clean, format convert and denoise the text data; image standardization, size normalization and content recognition are performed on the image data; key frame extraction and speech-to-text processing are performed on the video material. A unified material index is constructed, and is labeled according to the material type and time, source, etc. dimensions, facilitating subsequent retrieval and matching. Thus, comprehensive collection of multi-source heterogeneous information of the target person is realized, avoiding one-sided or incomplete information of the person caused by single data source.

[0034] In step S130, for each attribute description information of the person, at least one attribute matching material segment corresponding to the attribute description information of the person is extracted from the target multi-source material data, and each attribute matching material segment corresponding to the attribute description information of the person is fused based on a multi-modal large model to output corresponding attribute multi-modal generated data, and the multi-modal generated data includes text generated data, image generated data and video generated data.

[0035] In some embodiments, for each attribute description information of the person (such as "occupation: scientist"), a rule or a classification model is designed to extract the segments related to the attribute from the target multi-source material data, for example, in the text segment aspect, sentences containing keywords such as "science" and "research" are extracted, in the image segment aspect, pictures related to laboratory scenes and scientific research equipment are extracted, and in the video segment aspect, speech segments of the target person at academic conferences or experimental operation pictures are extracted.

[0036] Further, the extracted text segments, image segments, and video segments are input into a multi-modal large model (such as CLIP or BLIP-2) for cross-modal fusion. Specifically, in the contrast learning model of CLIP, it can map images and texts into a common embedding space, and through contrast learning, the distance between image-text pairs with similar semantics in the embedding space is closer, while the distance between dissimilar pairs is farther. In the present embodiment, an improved model structure based on CLIP is adopted. For the extracted text segments and image segments (or frame image sequences corresponding to video segments), the improved CLIP can well perform cross-modal fusion and learn the semantic association between cross-modal data, making up for the shortcomings of traditional text extraction in semantic understanding and context expression. More details of the improvement of the model will be described in conjunction with other examples below.

[0037] Through the embodiments of the present application, the pre-trained multi-modal large model is used to fuse and understand the content of images and texts / videos. Multi-modal fusion can complement the key information in each data source, making the expression of each attribute more comprehensive and the semantics more accurate, reducing the extraction errors caused by modal bias, and enhancing the multi-modal expression ability of the character attribute information, so that the abstract attributes have intuitive visual and behavioral expression.

[0038] In step S140, each attribute multi-modal generated data is filled into the corresponding attribute node in the knowledge graph framework according to the character attribute description information, to construct a multi-modal character knowledge graph.

[0039] In some embodiments, each attribute multi-modal generated data is mapped to the attribute node in the corresponding knowledge graph framework. For example, the multi-modal generated data (text, image, and video) of "occupation: scientist" is filled into the corresponding attribute node, and a knowledge graph visualization tool (such as Gephi or D3.js) is used to present the relationship between the core node and the attribute node in a graphical manner. Each attribute node is attached with a link to the corresponding multi-modal generated data, so that the user can view the specific content after clicking, improving the interactive experience. Thus, through the visualized multi-modal character knowledge graph, such as the Hunan culture knowledge graph centered on modern Hunan characters, the structure is logically clear, and each attribute node enriches the connotation of the character portrait, meeting the needs of fine-grained analysis.

[0040] Through the embodiments of the present application, a structured knowledge graph framework is constructed, efficient organization and retrieval of target person and attribute information are realized; multi-source data collection and semantic analysis technology is used to ensure the comprehensiveness and accurate matching of the material; with the help of multi-modal large model, text, image, video and other multi-source fragments are fused to generate high-quality and semantically consistent multi-modal data, through accurate matching and intelligent fusion, fine-grained extraction and semantic enhancement of each attribute of the person are realized; at the same time, through targeted retrieval and filling of the attribute nodes of the knowledge graph, a multi-modal person knowledge graph that is both structured and dynamically updated is accurately constructed, dynamic updating and visual display are supported, and the timeliness of information and user interaction experience are significantly improved.

[0041] Figure 2 An operation flow diagram is shown according to an example of step S110 in Figure 1 .

[0042] As shown in Figure 2 , in step S210, user input information is obtained, and a target person name is determined according to the user input information.

[0043] In some embodiments, the target person name is determined by user input information, and in addition, to improve accuracy, the input name can also be standardized (such as disambiguation, format unification, etc.), to exclude the confusion risk that may be caused by alias, abbreviation or same-name person.

[0044] In step S220, a person event recognition prompt word is constructed according to the target person name and a preset prompt word template, and the person event recognition prompt word is input into a large language model to trigger the large language model to output at least one person event keyword for the target person name.

[0045] Specifically, one or more prompt word templates for identifying person events are pre-set in the system to guide the large language model to generate keywords. Examples of prompt word templates are as follows:

[0046] "Please list the important events related to [target person name]."

[0047] "What are the main achievements of [target person name]?"

[0048] "Describe [target person name]'s contribution in [field / industry]."

[0049] Further, the target person name is dynamically inserted into the prompt word template to generate a personalized prompt word. For example, inputting "Zhang San" generates the prompt word "Please list important events related to Zhang San." The generated prompt word is input into a large language model (such as GPT or Qwen series) to trigger the output of keywords related to the target person. For example, taking "Yuan Longping", a representative figure of Huxiang, as the input example of the target person name, the keywords "hybrid rice", "three-line method", and "dream of taking a cool ride under the rice" may be generated. Thus, through the preset prompt word template and dynamic filling mechanism, high-quality event identification prompt words are quickly generated, the large language model is automatically triggered to output event keywords related to the target person, and the key events related to the person can be more accurately captured to determine more fine-grained and representative attribute description information of the person.

[0050] It should be noted that in the calling process, the temperature parameter, response length, and other configurations of the large language model can be set to ensure that the large language model can both explore details and ensure the accuracy of the output. With the semantic reasoning capability of the large language model, diverse event keywords of the person can be extracted from a wide knowledge base, and important event keywords related to the target person can be effectively extracted, which not only expands the background information of the person but also provides high-quality event clues for subsequent attribute mapping.

[0051] In step S230, the attribute description information of the person is determined according to each event keyword of the person.

[0052] In some embodiments, the attribute description information of the person can be directly defined according to the event keywords of the person, for example, "hybrid rice", "three-line method", and "dream of taking a cool ride under the rice" are respectively taken as information related to the corresponding attributes of the person to trigger the generation of exclusive attribute nodes of the corresponding major events of the person in the subsequent generation of the person knowledge graph, and the personalized person knowledge graph corresponding to different person names is realized.

[0053] Regarding the implementation details of step S230, in some examples of the embodiments of the present application, the customized attribute description information of the person is determined according to each event keyword of the person, and the attribute description information of the person is determined according to at least one preset general attribute description information of the person and each customized attribute description information of the person.

[0054] Through the embodiments of the present application, customized attribute description information of the person is generated for each event keyword, the system can capture the specific attribute characteristics of the target person in different event backgrounds, reflect personalized details, and the customized description is based on specific event keywords, which is closer to the actual context of the target person, and improves the accuracy of semantic matching. In addition, the combination of these customized information and the preset general attribute description information realizes standardized description, effectively balances personalization and universality, and ensures that the knowledge graph has high generalization ability while expressing in fine granularity.

[0055] Figure 3 An operation flow diagram of an example of step S120 in the method 1000 is shown. Figure 1 An operation flow diagram of an example of step S120 in the method 1000 is shown.

[0056] In step S310, the target person name is semantically matched with the material description information of each multi-source material in the multi-source material library to screen at least one initial matching multi-source material from the multi-source material library.

[0057] In some embodiments, the description information (including text title, label, abstract, etc.) of each material in the multi-source material library is subjected to word segmentation, stop word removal, and format normalization processing, and a pre-trained language model (such as BERT) is used to convert the text information into a high-dimensional semantic vector. Similarly, the target person name obtained and standardized is also subjected to the same semantic encoding to generate a semantic vector of the target name. Further, by using cosine similarity or other distance measurement methods, the similarity score between the target person name vector and each material description vector is calculated. According to the set similarity threshold, the materials with high similarity are screened out to ensure that at least one initial matching material is obtained, and at the same time, multiple candidate materials are sorted in descending order of similarity, which can greatly improve the accuracy and recall rate of preliminary screening.

[0058] In step S320, the data noise score of each initial matching multi-source material is determined based on a data noise scoring network, which includes a parallel text data noise evaluation branch, an image data noise evaluation branch, and a video data noise evaluation branch.

[0059] In the multi-modal data noise scoring network, the text data noise evaluation branch uses a noise detection model to determine whether the text content has syntax errors, spelling errors, or information redundancy, the image data noise evaluation branch uses an image quality evaluation algorithm (such as a quality evaluation based on a convolutional neural network) to detect image clarity, noise, distortion, and other problems, and the video data noise evaluation branch uses key frame extraction and video quality detection algorithms to evaluate whether there are noise factors such as blurring, frame loss, and sound interference in the video. Here, each modal scoring branch independently scores the data of the corresponding modal, outputs the noise scores of the text, image, and video, and aggregates them into a comprehensive noise score. In this way, through the parallel noise scoring network, the system can perform detailed and quantitative noise evaluation on multi-modal materials.

[0060] Each branch in the data noise scoring network can be trained using a self-supervised contrast learning strategy. By constructing contrast samples of clear data and noisy data, the network automatically learns the feature differences between high-quality and low-quality data, thereby improving the scoring accuracy.

[0061] In step S330, the initial matching multi-source material corresponding to the data noise score not exceeding the preset noise score threshold is determined as the first initial matching material, and the initial matching multi-source material corresponding to the data noise score exceeding the noise score threshold is determined as the second initial matching multi-source material.

[0062] In some embodiments, the noise scores independently output for each branch are respectively filtered according to preset noise level thresholds for text, image and video materials respectively. Through explicit noise thresholds and classification mechanisms, the system can automatically distinguish high-quality and low-quality materials, ensure that only high signal-to-noise ratio data directly participates in subsequent construction, and separately process low-quality data, thereby improving the overall accuracy and reliability of the final multi-source material data.

[0063] It should be noted that in some application scenarios, the noise scores output by each branch can be weighted and fused to obtain a comprehensive multi-modal noise score, but in the present embodiment, it is more recommended to maintain modality independence in order to adopt customized reconstruction strategies for different materials.

[0064] In step S340, the second initial matching multi-source material is input into a material reconstruction network to generate reconstructed multi-source material. The material reconstruction network includes a parallel text data reconstruction branch, an image data reconstruction branch and a video data reconstruction branch.

[0065] Here, the text reconstruction branch uses a text correction and generation model to perform spelling correction, grammar optimization, content completion or abstract generation on the original text, thereby generating text data with clearer semantics and more standardized expression. The image reconstruction branch uses image enhancement techniques such as super-resolution reconstruction, denoising processing and image repair algorithms to process low-quality images, restore or enhance image details and clarity. The video reconstruction branch combines key frame extraction, video denoising, picture quality enhancement and speech recognition optimization techniques to reconstruct low-quality videos and output smooth and clear video data.

[0066] Specifically, each branch in the material reconstruction network independently reconstructs and processes low-quality material data, and outputs reconstructed text, image and video data. Preferably, the reconstructed data can be noise scored again to ensure that the processed data meets the preset quality standards, and only the reconstructed data with a noise score not exceeding the threshold is selected as the effective output.

[0067] Through the material reconstruction network, the data quality of low-quality materials can be effectively repaired and improved. By reconstructing text, image and video data, noise interference is reduced, and even if the original data has noise problems, clear and accurate information can still be provided after reconstruction, providing high-quality support for knowledge graph construction.

[0068] In step S350 , target multi-source material data is determined according to the first initial matching material and the target reconstructed multi-source material, where the target reconstructed multi-source material is a reconstructed multi-source material whose corresponding noise score does not exceed a noise score threshold.

[0069] Specifically, on the one hand, the first initial matching material with a noise score that meets the requirements is directly incorporated into the target multi-source material data. On the other hand, the second initial matching material is processed by the reconstruction network, and the noise score is re-evaluated. The reconstructed data with a score that does not exceed the noise threshold is selected as the target reconstructed multi-source material. Thus, by integrating high-quality original data with reconstructed and enhanced data, the final target multi-source material data generated has a high signal-to-noise ratio, completeness, and diversity, ensuring that subsequent multimodal information fusion and knowledge graph construction can rely on accurate, rich, and clear data resources to effectively support the refined analysis and expression of character information.

[0070] Regarding the specific details of the data noise scoring network, in some examples of the embodiments of this application, the text data noise evaluation branch is used to extract the global semantic features of the text in the initial matching multi-source material through the BERT model, map the extracted text features through the fully connected layer and normalize them using the Sigmoid activation function to obtain the corresponding text noise score:

[0071] h T =BERT(T), formula (1)

[0072] s T =σ(W T ·h T +b T ), Formula (2)

[0073] Where h T Represents the feature vector obtained by the BERT model for the text T, s T ∈[0,1] represents the text noise score, W T and b T They represent the fully connected layer weight matrix and the fully connected layer bias term of the text scoring branch respectively; σ is the Sigmoid activation function, which is used to ensure that the output is normalized to [0,1].

[0074] It should be understood that a higher score value indicates more severe noise, and a lower score value indicates higher data quality.

[0075] In the text data noise evaluation branch, the global semantic features of the text material are extracted using a pre-trained language model (e.g., BERT), and the [CLS] vector of BERT is used as the text representation. The extracted text features are mapped to a scalar noise score through a fully connected layer, and normalized using a Sigmoid activation function, thereby obtaining the text noise score. In this way, spelling, grammar errors and ambiguity problems in the text can be effectively captured, thereby accurately judging the text data during noise filtering.

[0076] The image data noise evaluation branch is used to extract deep features of images in the initial matching multi-source material through a convolutional neural network. The feature maps of the convolutional layer are pooled into fixed-length feature vectors using global average pooling. The pooled image features are mapped using a fully connected layer and normalized using Sigmoid, thereby obtaining the image noise score:

[0077] F I = CNN(I), equation (3)

[0078] f I = GAP(F I ), equation (4)

[0079] s I = σ(W I · f I + b I ), equation (5)

[0080] where F I represents the feature map of image I extracted through the convolutional network, f I represents the image feature vector after global average pooling; CNN represents the convolutional neural network, and GAP represents the global average pooling; s I ∈ [0, 1] represents the image noise score, W I and b I represent the weight matrix and bias term of the fully connected layer of the image score branch, respectively.

[0081] In the image data noise evaluation branch, a pre-trained convolutional neural network is used to extract deep features of the image, and global average pooling (GAP) is used to pool the feature maps of the convolutional layer into fixed-length feature vectors, achieving global feature pooling. The pooled image features are mapped to a noise score using a fully connected layer, and the image noise score is output through Sigmoid normalization. In this way, problems such as blur and low resolution in the image can be accurately identified.

[0082] The video data noise evaluation branch is used to extract at least one first key frame of the video in the initial matching multi-source material, and input into the image data noise evaluation branch respectively to calculate the corresponding key frame noise score, and aggregate the scores of all first key frames by using a weighted average method, so as to obtain the video noise score:

[0083]

[0084] In the formula, f vj represents the global image feature vector of the jth first key frame of the video, s vj ∈[0,1] represents the noise score of the jth first key frame calculated by calling the image data noise evaluation branch, N v represents the total number of the extracted first key frames in the video, is a trainable parameter vector for calculating the importance of the first key frame, represents the transpose vector of , represents the sum of the importance score index values of the N v first key frames in the video, a j represents the importance weight of the jth first key frame, and satisfies s v ∈[0,1] represents the video noise score.

[0085] In the video data noise evaluation branch, a key frame extraction algorithm is used to select several representative frames from the video The image data noise score branch is applied to each key frame to independently calculate the noise score s vj of each frame, and the scores of all key frames are aggregated in time sequence by using a weighted average method to obtain the overall video noise score s v . In this way, the frames with higher noise in the video are given lower weights by the time sequence aggregation method, so as to obtain a stable noise score reflecting the overall video quality, which provides a basis for video data reconstruction and quality control. In addition, the image data noise score branch is directly reused to score each key frame, which effectively reduces the complexity and model data amount of the video data noise evaluation branch, and also reduces the demand for system running resources.

[0086] Through the design of the above data noise scoring network, the noise of text, image and video data is evaluated independently and accurately in different modes. Specifically, different branches are designed for different data modalities, which effectively captures syntax errors and ambiguities in text, blurred or low-quality features in images, and dynamic changes in key frames in videos. Thus, through the sub-modal noise evaluation method, the actual quality of each data type can be more accurately reflected. In addition, the attention weight mechanism introduced in the video branch ensures that the importance of each key frame can be adaptively adjusted, thereby obtaining a more stable and objective overall video noise score. This mechanism performs particularly well in complex video scenarios.

[0087] Regarding the implementation details of the material reconstruction network, in some examples of the embodiments of the present application, the text data reconstruction branch is used to adopt a pre-trained generation model as a conditional generator to generate reconstructed text from the input noisy text by maximizing the conditional generation probability:

[0088]

[0089] where T' is the input noisy text, representing the preprocessed text data; T cand is the candidate generated text, representing the text sequence that the generator can output; is the final generated reconstructed text, aiming to eliminate noise errors in the original text and restore semantics; P(T cand | T'; θ T ) is the conditional generation probability calculated by the pre-trained generation model, which represents the probability of generating text T T under model parameters θ cand , given the input T'.

[0090] In the text data reconstruction branch, the pre-trained generation model can adopt a T5 (Text-to-Text Transfer Transformer) model as the generator to achieve text repair through conditional generation, so that the output text has both high grammatical accuracy and the ability to restore key information. In addition, semantic consistency detection is used to ensure the consistency of the reconstructed text and the original text in terms of key information, thereby avoiding information deviation caused by excessive generation. As a result, the final output reconstructed text has higher accuracy and clarity, and can effectively remove spelling errors, grammatical problems and ambiguous expressions.

[0091] The image data reconstruction branch is used to adopt an image reconstruction generator based on a generative adversarial network to reconstruct the input noisy image to generate a reconstructed image:

[0092]

[0093] where I' is the input noisy image, representing the pre-processed image data; G(·; θ I ) is the image reconstruction generator, whose model parameters are θ I ; represents the final generated reconstructed image, aiming to repair the blur and noise in the original image; L I represents the total loss of the image reconstruction generator, including the reconstruction loss and the adversarial loss; represents the pixel-level L2 loss, used to measure the gap between the reconstructed image and the target reference image I ref ; I ref represents the reference image used for supervised training, L adv represents the adversarial loss output by the discriminator of the generative adversarial network; λ adv is the weight coefficient of the adversarial loss, used to balance the influence between the reconstruction loss and the adversarial loss.

[0094] In the image data reconstruction branch, the low-quality image is reconstructed by using the image generation network (such as the generative adversarial network), and the advantages of the generative model are used to repair the low-quality input by learning the distribution of a large number of high-quality images, analyze the problems such as blur, low contrast or noise in the image, automatically restore the details of the image, and generate a clear and detailed reconstructed image.

[0095] In addition, in the loss function of the image data reconstruction branch, the pixel-level reconstruction term and the adversarial term (λ adv L adv ) are introduced at the same time, the reconstruction term constrains the consistency of the reconstructed image and the reference image I ref at the pixel level, ensuring the accurate restoration of the global structure and details; the adversarial term uses the discriminator of the adversarial network to supervise the generator to output images with higher visual authenticity, thereby effectively avoiding the image "over-smoothing" or texture detail loss caused by simple pixel-level reconstruction. Therefore, in the image generation or reconstruction process, the main structure of the reference image can be accurately restored, and the visual details and realism can be improved through the adversarial learning mechanism.

[0096] The video data reconstruction branch is used to extract at least one second key frame from the input noisy video, and input the second key frame into the image reconstruction branch respectively to generate corresponding reconstructed key frames, and use the time series modeling to process the reconstructed key frames for smoothing and time series consistency, to generate a reconstructed video:

[0097]

[0098] wherein, represents I ireconstructed key frame generated after the image reconstruction generator, I i M represents the total number of the second key frames extracted from the input noisy video; h i and h i-1 respectively represent the hidden state representations of the i-th second key frame and the i-1-th second key frame in the LSTM time sequence model, θ V represents the model parameters of the LSTM time sequence model; is the final output reconstructed video, which represents a sequence of reconstructed video frames.

[0099] In the video data reconstruction branch, the method in the image reconstruction branch is separately used for reconstruction for each key frame, ensuring that the quality of each frame is improved, while the complexity and model data amount of the video data reconstruction branch are reduced through model structure reuse. In addition, since the video has a time sequence continuity requirement, after reconstruction, the LSTM time sequence model is further used to process all the key frames, smooth the inter-frame transition, correct the frame skipping problem caused by motion or changes, and ensure the overall coherence of the video, which meets the time sequence characteristics of the real video.

[0100] Through the embodiments of the present application, the collaborative work of each branch in the material reconstruction network is utilized to specially reconstruct the noise characteristics of different modalities (text, image, and video), so that the output data meets high standards in syntax, vision, and time sequence. In addition, through the reconstruction of different modalities, each branch adopts a targeted reconstruction strategy, and dynamic adaptive repair of different types of data noise is realized.

[0101] It should be noted that in the video branch of the data noise scoring network and the material reconstruction network described above, the extraction of key frames in the video is involved, and the type of the key frame extraction algorithm suitable for being used can be diversified. In some examples of the embodiments of the present application, the key frame extraction algorithm can be a combination of preliminary screening based on optical flow and color histogram difference and video deep semantic analysis, to realize the identification and extraction of key frames.

[0102] Specifically, in the preliminary screening based on the difference between optical flow and color histogram, in order to capture the frames with large scene or motion changes in the video, first, the difference between the optical flow and the color histogram of the continuous frames is detected.

[0103] 1) Optical flow change detection

[0104] For the continuous frames I t and, the optical flow field OF t is calculated, and the average optical flow amplitude is defined as:

[0105]

[0106] where M t represents the average optical flow magnitude between frame t and frame t-1, N pix represents the total number of pixels in the frame, OF t (i) represents the optical flow vector of the i-th pixel between frame t and frame t-1.

[0107] When M t exceeds a preset threshold τ flow , it is considered that frame t has a significant change and can be one of the candidate key frames.

[0108] 2) Color histogram difference

[0109] The color histograms H(I t ) and H(I t-1 ) of frames t and t-1 are calculated, and the difference is measured using the chi-square distance d t :

[0110] d t = χ 2 (H(I t ), H(I t-1 )), equation (14)

[0111] where d t represents the color histogram difference of frame t; χ 2 represents the chi-square distance function, and the larger the output value, the more obvious the color distribution change.

[0112] If d t > τ hist , frame t can also be marked as one of the candidate key frames.

[0113] Here, through the preliminary screening of optical flow and color histogram, the number of frames to be processed is reduced, thereby improving the efficiency of subsequent depth feature extraction and clustering.

[0114] In video depth semantic analysis, depth features are extracted from all candidate key frames in the video to obtain higher-level semantic depth features. According to the clustering of the semantic depth features of the candidate frames, different content modes in the video are distinguished, and the most representative frame is selected from each cluster as a key frame.

[0115] Specifically, the depth features of each frame are extracted using a pre-trained convolutional neural network:

[0116] r t = CNN(I t ), equation (15)

[0117] where r t represents the depth feature vector of frame t.

[0118] The K-means algorithm is used to cluster the features of all candidate frames. Let the number of clusters be Z, and the center of each cluster C z be:

[0119]

[0120] where μ z represents the center vector of the z-th cluster, and |C z | represents the number of frames in cluster z.

[0121] In each cluster C z , the frame closest to the cluster center is selected as the representative key frame of the cluster:

[0122]

[0123] where t z represents the index of the selected key frame in cluster z.

[0124] After K-means clustering, the frames selected from the Z clusters are taken as the key frame set.

[0125] Here, by using deep feature extraction and K-means clustering, the representative information of different visual content in the video can be accurately captured from the candidate frames, ensuring that the selected key frames can fully reflect the main changes in the video.

[0126] Through the embodiments of the present application, the key frame is defined as the frame of the main visual change and content in the video, and by combining the optical flow, color histogram, depth feature and temporal attention mechanism, the stability and adaptability of key frame extraction in various video scenes are improved. In addition, by dynamically setting the number of clusters Z and the temporal attention weight, the algorithm can adaptively adjust the number of key frames and the selection criteria according to the complexity of the video content, further improving the flexibility and data representativeness of the overall system.

[0127] Figure 4 An operation flow diagram of an example of extracting attribute matching material segments from target multi-source material data according to an embodiment of the present application is shown.

[0128] As shown in Figure 4 , in step S410, attribute tag groups are determined according to each character attribute description information, and a label classifier for the attribute tag groups is constructed, and each attribute tag is used to indicate the corresponding character attribute description information.

[0129] In some embodiments, based on the acquired character attribute description information (such as occupation, interest, event, etc.), the historical data and domain knowledge are statistically analyzed and analyzed, and the key attribute words are automatically extracted to form a preliminary attribute label list, such as "occupation: scientist", "nationality: China", "achievement: Nobel Prize", and a set of semantic attribute labels are generated.

[0130] Further, a label classifier for the attribute label set is constructed using machine learning or deep learning methods. Specifically, the material segments containing relevant attribute descriptions are extracted from the existing data, and the corresponding attribute labels are labeled, and a suitable classification model (such as BERT, TextCNN or SVM) is selected as the basic framework of the label classifier. Further, the labeled data set is used to train the label classifier, so that it can accurately match the corresponding attribute label according to the input semantic description.

[0131] In step S420, the target multi-source material data is semantically segmented to obtain a plurality of material candidate segments, each material candidate segment having a corresponding material segment semantic description.

[0132] In some embodiments, text segmentation can use sentence boundary detection and semantic topic segmentation techniques to divide long texts so that each paragraph or sentence has relatively independent semantic information. Image region segmentation can use target detection and image region segmentation algorithms to cut different regions (such as human faces, clothing, and scene backgrounds) in an image, ensuring that each region corresponds to an independent semantic description. Video scene division uses key frame extraction and scene change detection to segment videos, dividing parts with unified semantic content in continuous shots into independent segments while extracting corresponding audio text transcription information.

[0133] Regarding the details of generating semantic descriptions of material candidate segments, a pre-trained language model can be used to generate summaries for each text segment to extract core semantics; image recognition technology can be used to generate corresponding image labels or descriptions; and key frame images and audio transcription results can be combined to generate comprehensive semantic descriptions of video segments through multi-modal fusion.

[0134] In step S430, the attribute labels matched by each material segment semantic description are determined based on the label classifier, thereby obtaining attribute matching material segments of the corresponding character attribute description information.

[0135] In some embodiments, the semantic description vector of each candidate material segment is input to the label classifier, so that the label classifier outputs a corresponding matching label probability distribution for each material segment. Here, one candidate segment is allowed to match multiple attribute labels, fully reflecting the multiple character attribute information that may exist in the material. Further, the attribute label probabilities output by the label classifier are threshold filtered to determine which candidate segments meet the matching requirements of a specific attribute. Thus, based on the automatic matching of the label classifier, the efficiency and accuracy of accurately screening content related to character attributes from a large number of candidate materials are significantly improved. In addition, through multi-label matching, multi-dimensional attribute information in a candidate segment can be captured at the same time, ensuring the comprehensiveness and granularity of character attribute descriptions in the knowledge graph.

[0136] In the embodiments of the present application, standard or customized character attribute description information is regarded as an independent label to form a label set. Each label represents a character attribute (for example, "appearance", "occupation", "interests and hobbies", "experience", etc.). Based on this label set, a label classifier is constructed by using a pre-trained model or a deep neural network, which is used to judge whether an input material segment belongs to a certain attribute label. By converting the attribute matching task into a label classification task, the character attribute description information and the material segment are explicitly associated, thereby realizing automatic attribute matching.

[0137] Thus, by using the label classifier to regard the character attribute description information as an independent label, the segments highly related to each attribute can be accurately identified in multi-source materials, fine-grained attribute matching is realized, and the accuracy and comprehensiveness of the attribute information of the knowledge graph are significantly improved.

[0138] In some examples of the embodiments of the present application, the label classifier includes a cascaded projection layer, a label prompt fusion layer, and a label matching layer.

[0139] The projection layer is used to map the semantic description of the material segment to an intermediate representation space:

[0140] h1 = ReLU (W1x + b1), equation (18)

[0141] In the equation, x represents the embedded representation of the semantic description of the material segment, W1 and b1 represent the weight matrix and bias term of the projection layer respectively, h1 represents the projection intermediate representation, and ReLU represents the ReLU activation function.

[0142] In the projection layer, the system uses linear transformation and ReLU activation to map the semantic description of the candidate material segment to a feature space more suitable for subsequent processing, ensuring that the input data is normalized to a certain dimension before entering the next layer, and strengthening the attention to "useful" features and suppressing irrelevant or noise features.

[0143] The label prompt fusion layer is used to encode the semantic information of each attribute label into a prompt vector to perform feature alignment with the material segment semantic description in the semantic space:

[0144] h 2,c = ReLU (h1+ P c ), equation (19)

[0145]

[0146] In the equation, l c represents the c-th attribute label, P represents the label prompt matrix corresponding to the attribute label group, P c represents the learnable prompt vector corresponding to l c , h 2,c represents the fused feature representation of the material segment semantic description and l c , and n represents the total number of attribute labels in the attribute label group. H2 represents the label fusion feature matrix.

[0147] In the label prompt fusion layer, the system adds the intermediate representation of the candidate material segment to the prompt vector of a certain person attribute label, and then performs ReLU activation again to obtain the fused representation. The activation again can be regarded as a fusion process of "label semantics" and "segment semantics". In addition, the fused vectors are stacked by row or by column to form a matrix containing the fused representations of multiple person attribute labels, which provides structured data support for multi-label joint prediction.

[0148] It should be noted that, unlike a simple single-layer linear classifier, in the embodiments of the present application, the use of two ReLU activations and the fusion of the label prompt vector and the candidate segment feature significantly enhances the network's ability to capture the semantics of the person attribute label. While capturing the segment feature, the label prompt vector is used to conduct customized feature mining for each label, greatly improving the precision of attribute matching. In addition, through the learnable prompt vector, the semantic information of each label can be directly aligned with the segment in the feature space, which can quickly adapt to new attribute labels with a small amount of annotation or prompt tuning, and has stronger scalability and domain adaptability.

[0149] The label matching layer is used to calculate the matching score corresponding to each attribute label:

[0150]

[0151] In the equation, z represents the transpose vector of the shared score weight vector, b 2,c represents the bias term of the attribute label l c , and z c represents the semantic description of the material segment in l cthe unnormalized matching score.

[0152] After obtaining the fusion representation, the unnormalized matching scores of the candidate material segment and each label are calculated by linear transformation and bias term. The score vector (or matrix) maps the fusion representation to a distinguishable "matching score" space, indicating the relevance of the current segment to each attribute label.

[0153] Based on the label matching layer, each matching score is processed by Sigmoid activation to obtain the matching probability of the material segment semantic description for each corresponding attribute label:

[0154] p c = σ(z c ), equation (22)

[0155] In the formula, σ is the Sigmoid activation function, p c represents the matching probability between the material segment semantic description and l c .

[0156] Here, the matching score is normalized to the interval [0, 1] by the Sigmoid function, and the output value can be regarded as the confidence of "the candidate segment contains the attribute label", so that when predicting multiple labels, the probability is calculated for each label, and multiple labels can be activated at the same time.

[0157] According to the material candidate segment corresponding to the matching probability exceeding the preset probability threshold, the attribute matching material segment of the corresponding attribute label is determined.

[0158] Compared with the traditional method of relying only on keyword matching or single modal shallow method, the "segment feature" and "label prompt" are deeply fused in the unified semantic space by the embodiments of the present application, so that the character attribute information can be recognized more finely and accurately.

[0159] Regarding the implementation details of the multi-modal large model in step S130, in some embodiments, the multi-modal large model can use a pre-trained multi-modal model (for example, a CLIP model), and the multi-modal model can be fine-tuned to improve the accuracy and quality of the multi-modal character knowledge graph construction.

[0160] More specifically, the CLIP model itself has been pre-trained on a large scale and has strong generation or reconstruction capabilities. Here, the generation capability is not additionally fine-tuned, but more focus is placed on optimizing the fusion of multi-modal information and cross-modal consistency.

[0161] Specifically, the fine-tuning of the multi-modal large model includes:

[0162] Define the training data set:

[0163]

[0164] In the formula, denotes a training data set, N s denotes the total number of samples in the training data set, and u denotes the u-th specific sample in the data set; is the text segment corresponding to the u-th sample, and denotes the text description segment of the specific attribute of the character; is the image segment corresponding to the u-th sample, and denotes the image material segment of the specific attribute of the character; is the video segment corresponding to the u-th sample, and denotes the video material segment of the specific attribute of the character; y (u) denotes the true label of the specific attribute of the character corresponding to the u-th sample.

[0165] The embedding representations of each modality are obtained using a pre-trained multi-modal large model:

[0166]

[0167] In the formula, f T (·), f I (·) and f v (·) respectively denote a text encoder function, an image encoder function and a video encoder function, and are the embedding representations corresponding to the u-th sample text segment, the sample image segment and the sample video segment, respectively.

[0168] After passing through the cross-modal fusion module, a unified cross-modal feature representation is obtained:

[0169]

[0170] In the formula, denotes the cross-modal fusion feature corresponding to the u-th sample, and CrossAttention(·) denotes a cross-modal attention mechanism.

[0171] Here, the cross-modal attention mechanism (CrossAttention) flexibly captures the correlation and semantic complementary relationship of each modality feature, and fuses the embedding features of the three different modalities of text, image and video to obtain a unified semantic feature representation. In this way, efficient integration of text, image and video information is realized, and more accurate and comprehensive multi-modal feature representation is obtained.

[0172] The cross-modal consistency loss is calculated, including:

[0173] First, the cross-modal fusion feature is mapped into the independent embedding space of the text, image and video modalities, respectively, denoted as:

[0174]

[0175] wherein g T (·),g I (·),g v (·) respectively represent trainable network layers for mapping the fused cross-modal feature to the text modality space, the image modality space and the video modality space; and respectively represent the reconstructed text embedding vector, the reconstructed image embedding vector and the reconstructed video embedding vector.

[0176] Then, taking the real modality feature as a supervision signal, the consistency between the reconstructed feature and the actual feature is constrained:

[0177]

[0178] wherein represents the cross-modal consistency loss, which is used to ensure the consistency between the generated data and the real data.

[0179] Here, the core idea of the cross-modal consistency loss is to constrain the generated multi-modal feature (reconstructed feature) to be similar to the real single-modal feature, thereby significantly improving the semantic consistency of the cross-modal generated content, and ensuring that the fused feature can be reflected in the text, image and video modality spaces while maintaining consistency.

[0180] Specifically, in the training process combining the above formula (27), the real modality feature is used as a supervision signal to push the fused feature to be as close as possible to the real feature when mapped to a single modality space. Through the minimization of the distance of each modality feature, the model continuously optimizes the cross-modal consistency, so that the generated result is more coordinated and consistent in the semantic space. Thus, the semantic correlation and consistency between the multi-modal generated data (text, image, video) are significantly improved, effectively avoiding the semantic fragmentation or deviation of the generated content of each modality, and improving the overall quality of the generated data.

[0181] The multi-modal contrast loss is calculated, including:

[0182] In each training batch, text-image pairs and text-video pairs of the same attribute are selected as positive samples, and segments of different attributes are selected as negative samples, and then the cosine similarity between the embedding representations is calculated to optimize the cross-modal matching degree:

[0183]

[0184] wherein represents the multi-modal contrast loss, and sim(·) represents a vector similarity function, represents the vector similarity of the u-th text-image positive sample pair, a vector similarity representing the u-th text-video positive sample pair, a sum of similarities between the u-th text segment and other non-corresponding image segments of the person attribute, a sum of similarities between the u-th text segment and other non-corresponding video segments of the person attribute.

[0185] Here, the core of the design of the multi-modal contrast loss is to narrow the distance between the same attribute data of different modalities, while widening the distance between the modal data of different attributes. By using the contrast learning principle, the text-image pair and the text-video pair are trained as positive and negative samples to promote the model to learn high-quality cross-modal alignment feature space.

[0186] In the training process combining the above formula (28), each training batch contains a large number of positive and negative sample pairs. By calculating the cosine similarity, the optimization of the distance between positive and negative samples is supervised, and the contrast learning mechanism is used to ensure the high-quality alignment of feature embedding in the cross-modal space, providing a solid and reliable basic feature space for the generation and use of cross-modal fusion features.

[0187] According to the cross-modal consistency loss and the multi-modal contrast loss, the total loss function is calculated:

[0188]

[0189] In the formula, Ltotal represents the total loss function, and a, β are hyperparameter weights for controlling the relative importance of the cross-modal consistency loss and the multi-modal contrast loss.

[0190] The comprehensive loss function provided by the embodiments of the present application takes into account the two key requirements of multi-modal consistency and multi-modal feature alignment, and comprehensively optimizes the effect of cross-modal fusion and consistency generation. In addition, by adjusting the balance between the losses through the weight parameters a, β, the training process is more targeted, for example, by setting a = 0.6, β = 0.4 to highlight the importance of cross-modal consistency, while maintaining the cross-modal feature alignment effect. Thus, it effectively ensures that the generated modal data after cross-modal feature fusion is semantically unified in content and mutually corroborates, avoids mismatching of information between modalities, improves the quality and consistency of multi-modal information fusion, and realizes higher quality, multi-modal consistent generated data, and improves the accuracy and robustness of the final person knowledge graph.

[0191] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of actions, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application. In the above embodiments, the description of each embodiment is focused on, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0192] Figure 5 A structural block diagram of an example of a multi-modal large model driven person knowledge graph construction system according to an embodiment of the present application is shown.

[0193] As shown in Figure 5 The multi-modal large model driven person knowledge graph construction system 500 includes a graph framework construction unit 510, a multi-source material acquisition unit 520, an attribute data generation unit 530, and a graph attribute filling unit 540.

[0194] The graph framework construction unit 510 is configured to acquire a target person name and multi-person attribute description information, and construct a knowledge graph framework; the knowledge graph framework includes a core node and a plurality of attribute nodes connected to the core node, the core node is used to indicate the target person name, and each attribute node is used to indicate corresponding person attribute description information.

[0195] The multi-source material acquisition unit 520 is configured to acquire corresponding target multi-source material data according to the target person name, and the target multi-source material data includes text material, image material and video material matched with the target person name.

[0196] The attribute data generation unit 530 is configured to, for each of the person attribute description information, extract at least one attribute matching material segment corresponding to the person attribute description information from the target multi-source material data, and fuse each attribute matching material segment corresponding to the person attribute description information based on a multi-modal large model to output corresponding attribute multi-modal generation data; the multi-modal generation data includes text generation data, image generation data and video generation data.

[0197] The graph attribute filling unit 540 is configured to fill each of the attribute multi-modal generation data into the corresponding attribute node in the knowledge graph framework according to the person attribute description information, to construct a multi-modal person knowledge graph.

[0198] In some embodiments, the embodiments of the present application provide a non-volatile computer readable storage medium, wherein one or more programs including execution instructions are stored in the storage medium, and the execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to perform the steps of any one of the above multi-modal large model driven character knowledge graph construction methods.

[0199] In some embodiments, the embodiments of the present application also provide a computer program product, which includes a computer program stored on a non-volatile computer readable storage medium, and the computer program includes program instructions, which, when executed by a computer, cause the computer to perform the steps of any one of the above multi-modal large model driven character knowledge graph construction methods.

[0200] In some embodiments, the embodiments of the present application also provide an electronic device, which includes at least one processor and a memory connected to the at least one processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the multi-modal large model driven character knowledge graph construction method.

[0201] Figure 6 is a hardware structure schematic diagram of an electronic device for executing the multi-modal large model driven character knowledge graph construction method provided by another embodiment of the present application, as shown in Figure 6 The device includes:

[0202] one or more processors 610 and a memory 620, Figure 6 In an example, the processor 610 is taken as an example.

[0203] The device for executing the multi-modal large model driven character knowledge graph construction method can also include an input device 630 and an output device 640.

[0204] The processor 610, the memory 620, the input device 630 and the output device 640 can be connected through a bus or other means, Figure 6 In an example, the connection through the bus is taken as an example.

[0205] Memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the multimodal large model-driven method for constructing a person knowledge graph in the embodiments of the present application. Processor 610 executes the non-volatile software programs, instructions, and modules stored in memory 620 to execute various server functional applications and data processing, thereby implementing the multimodal large model-driven method for constructing a person knowledge graph in the above-mentioned method embodiment.

[0206] The memory 620 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 620 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include a memory remotely located relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0207] The input device 630 may receive input digital or character information and generate signals related to user settings and function control of the electronic device. The output device 640 may include a display device such as a display screen.

[0208] The one or more modules are stored in the memory 620, and when executed by the one or more processors 610, perform the multimodal large model driven character knowledge graph construction method in any of the above method embodiments.

[0209] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0210] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0211] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and their primary purpose is to provide voice and data communications. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0212] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers and have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPCs.

[0213] (3) Portable entertainment device: This kind of device can display and play multimedia content. This kind of device includes: audio, video player, palm game machine, electronic book, and smart toy and portable vehicle navigation device.

[0214] (4) Other on-board electronic devices with data interaction function, such as vehicle-mounted device installed on vehicle.

[0215] The device embodiments described above are only illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0216] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software plus a general hardware platform, and of course, it can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the methods described in each embodiment or some parts of the embodiment.

[0217] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a character knowledge graph driven by a multimodal large model, comprising: Obtain the target person's name and multiple character attribute descriptions, and build a knowledge graph framework; The knowledge graph framework includes a core node and a plurality of attribute nodes respectively connected to the core node, wherein the core node is used to indicate the name of the target person, and each attribute node is used to indicate corresponding person attribute description information; Acquire corresponding target multi-source material data according to the target person's name, wherein the target multi-source material data includes text material, image material, and video material that match the target person's name; For each of the character attribute description information, extract at least one attribute matching material segment corresponding to the character attribute description information from the target multi-source material data, and fuse the attribute matching material segments corresponding to the character attribute description information based on the multimodal large model to output corresponding attribute multimodal generated data; The multimodal generated data includes text generated data, image generated data and video generated data; Filling each of the attribute multimodal generated data into the corresponding attribute node in the knowledge graph framework according to the character attribute description information to construct a multimodal character knowledge graph; The step of extracting at least one attribute matching material segment corresponding to each character attribute description information from the target multi-source material data includes: Determining an attribute label group according to each of the character attribute description information, and constructing a label classifier for the attribute label group; each of the attribute labels is used to indicate the corresponding character attribute description information; semantically segmenting the target multi-source material data to obtain a plurality of candidate material segments, each of the candidate material segments having a corresponding material segment semantic description; Determining attribute tags that match the semantic descriptions of the respective material clips based on the tag classifier, thereby obtaining attribute-matched material clips corresponding to the character attribute description information; The label classifier comprises a cascaded projection layer, a label prompt fusion layer and a label matching layer; The projection layer is used to map the semantic description of the material clip into an intermediate representation space: ， Where, Embedded representation of the semantic description of the material clip, and Represent the weight matrix and bias term of the projection layer respectively, represents the projected intermediate representation, Represents the RELU activation function; The label hint fusion layer is used to encode the semantic information of each attribute label into a hint vector to align the features with the semantic description of the material clip in the semantic space: ， ， Where, express The corresponding learnable hint vector, Indicates the attribute tags, Represents the label prompt matrix corresponding to the attribute label group; Represents the semantic description of the material clip and The fused feature representation is Indicates the total number of attribute tags in the attribute tag group. Represents the label fusion feature matrix; The matching scores corresponding to each attribute label are calculated based on the label matching layer: ， Where, represents the transposed vector of the shared scoring weight vector, Indicates attribute tags The bias term, Indicates the semantic description of the material clip in Unnormalized matching score on ; Based on the label matching layer, each matching score is processed by Sigmoid activation to obtain the matching probability of the material segment semantic description for each corresponding attribute label: ， Where, is the Sigmoid activation function, Represents the semantic description of the material clip and The matching probability between According to the candidate material segments whose corresponding matching probabilities exceed a preset probability threshold, attribute matching material segments with corresponding attribute tags are determined.

2. The method according to claim 1, wherein The step of obtaining the target person's name and multiple descriptions of the person's attributes includes: Obtaining user input information, and determining the target person's name based on the user input information; Constructing a character event recognition prompt word based on the target character name and a preset prompt word template, and inputting the character event recognition prompt word into a large language model to trigger the large language model to output at least one character event keyword for the target character name; According to each of the character event keywords, character attribute description information is determined.

3. The method according to claim 2, wherein: Determining character attribute description information based on each character event keyword includes: Determining corresponding customized character attribute description information according to each of the character event keywords; The character attribute description information is determined according to at least one preset universal character attribute description information and each of the customized character attribute description information.

4. The method according to claim 1, wherein The step of obtaining corresponding target multi-source material data according to the target person's name includes: Performing semantic matching on the target person's name and the material description information of each multi-source material in the multi-source material library to select at least one initial matching multi-source material from the multi-source material library; Determining a data noise score for each of the initially matched multi-source materials based on a data noise scoring network; the data noise scoring network comprises a text data noise evaluation branch, an image data noise evaluation branch, and a video data noise evaluation branch connected in parallel; Determining the initial matching multi-source material whose corresponding data noise score does not exceed a preset noise score threshold as a first initial matching material, and determining the initial matching multi-source material whose corresponding data noise score exceeds the noise score threshold as a second initial matching multi-source material; Inputting the second initial matching multi-source material into a material reconstruction network to generate a reconstructed multi-source material; the material reconstruction network comprises a text data reconstruction branch, an image data reconstruction branch, and a video data reconstruction branch connected in parallel; Target multi-source material data is determined according to the first initial matching material and the target reconstructed multi-source material; the target reconstructed multi-source material is a reconstructed multi-source material whose corresponding noise score does not exceed the noise score threshold.

5. The method according to claim 1, wherein Fine-tuning training for the multimodal large model includes: Define the training dataset: ， Where, represents the training dataset, represents the total number of samples in the training dataset, Indicates the first A specific sample; For the The text fragment corresponding to each sample represents a text description fragment of a person's specific attributes; For the The image fragment corresponding to each sample represents the image material fragment with specific attributes of the person; For the The video clip corresponding to each sample represents the video clip with specific attributes of the person; Indicates the The true label of the person attribute corresponding to each sample; Use the pre-trained multimodal large model to obtain the embedded representation of each modality: ， Where, 、 and Represent the text encoder function, image encoder function and video encoder function respectively, 、 and Respectively The embedding representations corresponding to sample text segments, sample image segments, and sample video segments; After the cross-modal fusion module, a unified cross-modal feature representation is obtained: ， Where, Indicates the The cross-modal fusion features corresponding to the samples, represents the cross-modal attention mechanism; Calculate cross-modal consistency loss, including: First, cross-modal fusion features Mapped into independent embedding spaces for text, image, and video modalities, respectively, denoted as: ， Where, They represent the trainable network layers that map the fused cross-modal features to the text modality space, image modality space, and video modality space respectively; 、 and Represent the reconstructed text embedding vector, the reconstructed image embedding vector, and the reconstructed video embedding vector respectively; Then, the true modal features are used as supervisory signals to constrain the consistency between the reconstructed features and the actual features: ， Where, represents the cross-modal consistency loss, which is used to ensure the consistency between generated data and real data; Compute multimodal contrastive loss, including: In each training batch, text-image pairs and text-video pairs with the same attributes are selected as positive samples, and clips with different attributes are selected as negative samples. The cosine similarity between the embedding representations is then calculated to optimize the cross-modal matching degree: ， Where, represents the multimodal contrast loss, represents the vector similarity function, Indicates the The vector similarity of text-image positive sample pairs, Indicates the The vector similarity of text-video positive sample pairs, Indicates the The sum of similarities between text segments and other image segments that do not correspond to person attributes, Indicates the The sum of similarities between the text segments and other video segments with non-corresponding person attributes; According to the cross-modal consistency loss and the multimodal contrast loss, the total loss function is calculated: ， Where, represents the total loss function, is a hyperparameter weight used to control the relative importance of cross-modal consistency loss and multimodal contrast loss.

6. A multimodal large model-driven character knowledge graph construction system, configured to implement the multimodal large model-driven character knowledge graph construction method according to any one of claims 1 to 5; the system comprising: A graph framework construction unit is used to obtain the target person's name and multiple person attribute description information, and to construct a knowledge graph framework; The knowledge graph framework includes a core node and a plurality of attribute nodes respectively connected to the core node, wherein the core node is used to indicate the name of the target person, and each attribute node is used to indicate corresponding person attribute description information; a multi-source material acquisition unit, configured to acquire corresponding target multi-source material data according to the target person's name, wherein the target multi-source material data includes text materials, image materials, and video materials that match the target person's name; an attribute data generating unit, configured to extract, for each of the character attribute description information, at least one attribute matching material segment corresponding to the character attribute description information from the target multi-source material data, and fuse the attribute matching material segments corresponding to the character attribute description information based on the multimodal macro model to output corresponding attribute multimodal generated data; The multimodal generated data includes text generated data, image generated data and video generated data; The graph attribute filling unit is used to fill each of the attribute multimodal generated data into the corresponding attribute node in the knowledge graph framework according to the character attribute description information to construct a multimodal character knowledge graph.

Images