Multimodal knowledge base generation method, multimodal information retrieval enhanced generation method and digital human multimodal interaction method

Abstract

The application provides a multimodal knowledge base generation method, a multimodal information retrieval enhancement generation method and a digital person multimodal interaction method, relates to the technical field of multimodal interaction, and the method fuses the multimodal tags of text content and multimodal information by analyzing the text content and the multimodal information in a target file, obtains knowledge cards in a knowledge splitting manner, and then generates a multimodal knowledge base, so that the knowledge in the target file is split, facilitating subsequent knowledge retrieval, providing knowledge reserves for a large language model, improving the accuracy and reliability of the response content provided by the large language model, and reducing the probability of AI illusion. Moreover, since the knowledge cards contain multimodal tags, response content containing multimodal information can be generated, thereby enriching the diversity of response content and improving the user's interaction experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-modal interaction, in particular to a multi-modal knowledge base generation method, a multi-modal information retrieval enhancement generation method and a digital human multi-modal interaction method. BACKGROUND

[0002] Virtual digital human (referred to as digital human) refers to a digital character image close to human image created by using digital technology, which has specific appearance, gender and personality characteristics, and the ability to express by using language, facial expression and body movement. Digital human is divided into interactive digital human and non-interactive digital human according to whether it can interact with the outside world. Interactive digital human has the ability to recognize the outside environment and interact with users. Non-interactive digital human does not have the ability to recognize the outside environment and interact with users.

[0003] Interactive digital human can be responsible for job consultation, front desk reception, business guidance, intelligent customer service and other positions. Users can express their needs or questions to digital human through voice or text form through human-computer interaction page. Digital human generates response content through pre-set response mechanism and feeds back to users. Users can express further needs or questions according to the feedback response content, and digital human continues to feed back until the user's needs or questions are met or solved, and the current communication and interaction is ended. In the above communication and interaction process, the pre-set response mechanism is the key to the response ability of digital human. That is, how to use the user input voice or text to generate high-quality response content is the core ability of interactive digital human.

[0004] However, in the prior art, one of the pre-set response mechanisms is to pre-set several common questions and corresponding response contents. Users can only ask the same or similar questions as the common questions to give accurate response contents. The second pre-set response mechanism is to apply a professional large language model for response. Although this scheme can provide accurate response contents for users with a high probability, the professional large language model is obtained by fine-tuning a general large language model using a large amount of high-quality professional data, which has a high cost. Moreover, the professional large language model has the problem of AI illusion, which cannot guarantee the authenticity and reliability of the response content.

[0005] In addition, the professional large language model only supports text output, that is, pure text content is taken as the response content of the professional large language model. The form of the response content is single and cannot meet the needs of business scenarios. SUMMARY

[0006] This invention provides a method for generating a multimodal knowledge base, a method for enhancing multimodal information retrieval, and a method for multimodal interaction of digital humans, in order to address the deficiencies existing in related technologies.

[0007] This invention provides a method for generating a multimodal knowledge base, comprising:

[0008] Obtain the target file;

[0009] The target file is parsed to obtain the parsing result; the parsing result includes the text content and multimodal information in the target file;

[0010] Determine the multimodal tags of the multimodal information, and fuse the text content with the multimodal tags to obtain a fusion result;

[0011] The fusion result is split into knowledge segments to form at least one knowledge card, and a multimodal knowledge base is generated based on at least one of the knowledge cards.

[0012] According to a multimodal knowledge base generation method provided by the present invention, the step of parsing the target file to obtain the parsing result includes:

[0013] Determine the access address of the multimodal information;

[0014] Based on the multimodal tags and the access addresses, a mapping table for the multimodal information is constructed.

[0015] According to a method for generating a multimodal knowledge base provided by the present invention, the step of generating a multimodal knowledge base based on at least one of the knowledge cards, followed by:

[0016] If a knowledge base access request is received, the multimodal label in the first knowledge card corresponding to the knowledge base access request is replaced with the corresponding access address based on the mapping table to obtain the access result, and the access result is displayed on the access interface.

[0017] According to a method for generating a multimodal knowledge base provided by the present invention, the step of generating a multimodal knowledge base based on at least one of the knowledge cards further includes:

[0018] Receive an edit request for the second knowledge card in the multimodal knowledge base;

[0019] Based on the edit request, the second knowledge card is edited.

[0020] According to a multimodal knowledge base generation method provided by the present invention, the step of editing the second knowledge card based on the editing request includes:

[0021] If the editing request is to add target multimodal information, then determine the target multimodal tag and target access address of the target multimodal information;

[0022] The target multimodal tag and the target access address are stored in the mapping table, and the target multimodal tag is added to the second knowledge card.

[0023] According to a method for generating a multimodal knowledge base provided by the present invention, the step of constructing a mapping table of the multimodal information based on the multimodal tags and the access addresses includes:

[0024] Extract a summary of the multimodal information;

[0025] The mapping table is constructed based on the multimodal tag, the access address, and the digest.

[0026] According to a multimodal knowledge base generation method provided by the present invention, the text content includes content fragments, and the content fragments are determined based on the file type of the target file.

[0027] According to a multimodal knowledge base generation method provided by the present invention, the step of determining the content fragment includes:

[0028] In response to the file type being a text file, the text content of the text file is treated as a content fragment;

[0029] In response to the file type being a demo file, the text content of the demo file is segmented into one or more content fragments based on the page content of the demo file.

[0030] In response to the file type being a lightweight text file, the text content of the lightweight text file is segmented into one or more content fragments according to the minimum heading of the lightweight text file;

[0031] In response to the file type being a data file, the text content of the data file is segmented into one or more content fragments based on the layout format of the data file.

[0032] According to the multimodal knowledge base generation method provided by the present invention, the data file is specifically a table file;

[0033] The method of dividing the text content of the data class file into one or more content segments based on the layout format of the data class file includes:

[0034] If the first row of the table file is the header row, then each row in the table file other than the first row is treated as a content segment, and the column content in each content segment is determined to include the column header in the header row;

[0035] If the first row of the table file is not a header row, then each row in the table file will be treated as a content segment.

[0036] According to a multimodal knowledge base generation method provided by the present invention, the step of splitting the fusion result into knowledge cards to form at least one knowledge card includes:

[0037] Determine the split delimiter and knowledge card length threshold;

[0038] Based on the splitting delimiter and the knowledge card length threshold, the fusion result is split into a first set of knowledge fragments;

[0039] Based on the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold, the first knowledge fragment set is applied to obtain at least one knowledge card.

[0040] According to a multimodal knowledge base generation method provided by the present invention, the step of applying the first knowledge fragment set to obtain at least one knowledge card based on the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold includes:

[0041] For any knowledge fragment in the first set of knowledge fragments, if the length of any knowledge fragment does not exceed the knowledge card length threshold, then any knowledge fragment is treated as a knowledge card.

[0042] If the length of any knowledge fragment exceeds the knowledge card length threshold, then the knowledge fragment is split into multiple first knowledge segments with a length not exceeding the knowledge card length threshold, and each first knowledge segment is used as a knowledge card.

[0043] According to a multimodal knowledge base generation method provided by the present invention, the step of splitting any knowledge fragment into multiple first knowledge segments with a length not exceeding the knowledge card length threshold includes:

[0044] Break any of the knowledge fragments into multiple statements;

[0045] Based on the length of each statement and the knowledge card length threshold, multiple statement sets are determined from the multiple statements in statement order, and each statement set is used as a knowledge card;

[0046] The length of each set of statements does not exceed the knowledge card length threshold.

[0047] According to a multimodal knowledge base generation method provided by the present invention, the step of splitting the fusion result into knowledge cards to form at least one knowledge card further includes:

[0048] Determine a knowledge redundancy length threshold, and based on the current knowledge card, apply the knowledge redundancy length threshold to determine redundant knowledge;

[0049] The redundant knowledge is copied to the starting position of the next knowledge card after the current knowledge card.

[0050] According to a multimodal knowledge base generation method provided by the present invention, the step of determining redundant knowledge based on the current knowledge card and applying the knowledge redundancy length threshold includes:

[0051] Determine whether there is a second knowledge segment at the end of the current knowledge card that does not exceed the knowledge redundancy length threshold;

[0052] If the second knowledge fragment does not exist at the end of the current knowledge card, then the last statement of the current knowledge card is taken as the redundant knowledge.

[0053] If the second knowledge fragment exists at the end of the current knowledge card, then the redundant knowledge is determined based on the length of the second knowledge fragment and the length of the next knowledge card after the current knowledge card.

[0054] According to a multimodal knowledge base generation method provided by the present invention, the step of splitting the fusion result into knowledge cards to form at least one knowledge card further includes:

[0055] Extract the title information and topic transition points from the fusion result;

[0056] Based on the title information and the topic shift point, the fusion result is divided into a second set of knowledge fragments;

[0057] Based on the title information and the second set of knowledge fragments, at least one of the knowledge cards is determined.

[0058] According to a multimodal knowledge base generation method provided by the present invention, the step of splitting the fusion result into a second set of knowledge fragments based on the title information and the topic transition points includes:

[0059] Based on the title information, determine the initial set of knowledge fragments for the fusion result;

[0060] If it is determined that there is a specified knowledge fragment in the initial knowledge fragment set whose length exceeds the first preset length, and there is a topic transfer point in the specified knowledge fragment, then the specified knowledge fragment is segmented to obtain the segmentation result, and the non-specified knowledge fragments in the initial knowledge fragment set and the segmentation result are merged to obtain an intermediate knowledge fragment set.

[0061] If it is determined that there are two adjacent knowledge fragments with the same title level in the intermediate knowledge fragment set, and the length of at least one of the two adjacent knowledge fragments is less than the second preset length, then the two adjacent knowledge fragments are merged to obtain the second knowledge fragment set.

[0062] According to a method for generating a multimodal knowledge base provided by the present invention, the step of generating a multimodal knowledge base based on at least one of the knowledge cards further includes:

[0063] If a sharing request for a third knowledge card is received, determine the target account corresponding to the sharing request;

[0064] In response to the target account granting the sharing request, the third knowledge card is generated in the target account's multimodal knowledge base.

[0065] This invention also provides a multimodal information retrieval enhancement generation method, comprising:

[0066] Obtain interactive information;

[0067] Extract the semantic features of the interaction information, and based on the semantic features, determine the target knowledge card in the multimodal knowledge base that matches the semantic features of the interaction information;

[0068] Based on the target knowledge card, a large language model is applied to generate the response content corresponding to the interactive information;

[0069] The multimodal knowledge base is generated based on the multimodal knowledge base generation method described above.

[0070] According to a multimodal information retrieval enhancement generation method provided by the present invention, the step of determining target knowledge cards in a multimodal knowledge base that match the semantic features of the interaction information based on the semantic features includes:

[0071] Determine the knowledge vector of each knowledge card in the multimodal knowledge base;

[0072] The target knowledge card is determined based on the similarity between the semantic features and the knowledge vector.

[0073] According to a multimodal information retrieval enhancement generation method provided by the present invention, the response content includes a target multimodal tag, and the target multimodal tag corresponds to a target access address; the step of generating the response content corresponding to the interaction information based on the target knowledge card and applying a large language model includes:

[0074] Based on the target access address, the target multimodal information in the response content is determined, and the response content is displayed on the interactive interface.

[0075] This invention also provides a digital human multimodal interaction method, comprising:

[0076] Based on the above-described multimodal information retrieval enhancement generation method, the response content is determined;

[0077] Based on the response content, digital human driving data is generated; the digital human driving data is used to drive the digital human to respond to the interactive information.

[0078] The present invention also provides a modal knowledge base generation apparatus, comprising:

[0079] The file acquisition module is used to acquire the target file;

[0080] The file parsing module is used to parse the target file and obtain the parsing result; the parsing result includes the text content and multimodal information in the target file;

[0081] The fusion module is used to determine the multimodal tags of the multimodal information and fuse the text content with the multimodal tags to obtain a fusion result;

[0082] The knowledge splitting module is used to split the fusion result into knowledge cards to form at least one knowledge card, and generate a multimodal knowledge base based on at least one knowledge card.

[0083] The present invention also provides a multimodal information retrieval enhancement generation device, comprising:

[0084] The interaction module is used to obtain interaction information;

[0085] The matching module is used to extract the semantic features of the interaction information and, based on the semantic features, determine the target knowledge card in the multimodal knowledge base that matches the semantic features of the interaction information.

[0086] The response content generation module is used to generate response content corresponding to the interactive information based on the target knowledge card and by applying a large language model.

[0087] The multimodal knowledge base is generated based on the multimodal knowledge base generation method described above.

[0088] The present invention also provides a digital human multimodal interaction system, comprising:

[0089] The response content determination module is used to determine the response content based on the multimodal information retrieval enhancement generation method described above.

[0090] A digital human driving module is used to generate digital human driving data based on the response content; the digital human driving data is used to drive the digital human to respond to the interactive information.

[0091] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal knowledge base generation method, or the multimodal information retrieval enhancement generation method, or the digital human multimodal interaction method as described above.

[0092] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal knowledge base generation method, or the multimodal information retrieval enhancement generation method, or the digital human multimodal interaction method as described above.

[0093] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multimodal knowledge base generation method, the multimodal information retrieval enhancement generation method, or the digital human multimodal interaction method as described above.

[0094] The multimodal knowledge base generation method, multimodal information retrieval enhancement generation method, and digital human multimodal interaction method provided by this invention first obtain a target file; then, the target file is parsed to obtain a parsing result, which includes the text content and multimodal information in the target file; subsequently, the multimodal tags of the multimodal information are determined, and the text content and multimodal tags are fused to obtain a fusion result; finally, the fusion result is knowledge-splitting to form at least one knowledge card, and the knowledge cards are used to construct a multimodal knowledge base. This method, by parsing the text content and multimodal information in the target file and fusing the text content with the multimodal tags of the multimodal information, obtains knowledge cards through knowledge splitting, thereby generating a multimodal knowledge base. This splits the knowledge in the target file, facilitating subsequent knowledge retrieval, providing knowledge reserves for large language models, improving the accuracy and reliability of the response content provided by the large language model, and reducing the probability of AI illusions. Furthermore, since the knowledge cards contain multimodal tags, response content containing multimodal information can be generated, thereby enriching the diversity of response content and improving the user's interactive experience. Attached Figure Description

[0095] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is one of the flowcharts illustrating the multimodal knowledge base generation method provided by this invention.

[0097] Figure 2 This is a schematic diagram of a page of PPT content in the multimodal knowledge base generation method provided by the present invention.

[0098] Figure 3 This is a flowchart illustrating the multimodal information retrieval enhancement generation method provided by the present invention.

[0099] Figure 4 This is the second flowchart of the multimodal knowledge base generation method provided by the present invention.

[0100] Figure 5 This is a flowchart illustrating the digital human multimodal interaction method provided by the present invention.

[0101] Figure 6 This is a schematic diagram of the modal knowledge base generation device provided by the present invention.

[0102] Figure 7 This is a schematic diagram of the structure of the multimodal information retrieval enhancement generation device provided by the present invention.

[0103] Figure 8 This is a schematic diagram of the structure of digital human multimodal interaction provided by the present invention.

[0104] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0106] Because existing interactive digital humans typically employ pre-set response mechanisms in their communication processes, one approach involves pre-setting several common questions (FAQs) and corresponding responses. One implementation displays these FQs on the user interface, allowing users to select a question from them. The system then provides the corresponding response based on the user's choice. If a user's needs or questions do not fall under the pre-set FQs, they can opt for human assistance via remote access. However, this approach limits users to asking only a single FQ, preventing personalized questions, and the interface usually only displays a dozen or so FQs, making it suitable only for simple business scenarios.

[0107] Another approach involves displaying several frequently asked questions (FAQs) on the user interface and allowing users to personalize their input with voice or text. The system recognizes and analyzes the user's input, identifies the closest FQ based on keywords, and provides the corresponding response. If the response doesn't match the user's needs or questions, a human operator can be selected for remote assistance. While this approach allows users to ask personalized questions, the responses are still pre-set and rely on keywords in the user's input to determine the closest FQ and response, which can easily lead to irrelevant answers.

[0108] Although this solution can pre-set multimodal information as response content to achieve the technical effect of multimodal response, since the response content is pre-bound to a certain common question, there is no essential difference in technical implementation from plain text response. It cannot answer questions that exceed the scope of the pre-set question bank, nor can it flexibly organize the multimodal information required by the user based on their specific needs.

[0109] The second pre-set response mechanism involves fine-tuning the general-purpose language model using high-quality professional data to obtain a specialized language model, and then applying this specialized language model to provide responses. While this approach can provide users with accurate responses with a relatively high probability, the cost of fine-tuning the general-purpose language model is high. Moreover, specialized language models inevitably suffer from the illusion of artificial intelligence (AI), and cannot guarantee the authenticity and reliability of the response content.

[0110] Furthermore, the professional large language model only supports text output, meaning that plain text content is used as the response content of the professional large language model, and it cannot directly output multimodal information as the response content.

[0111] Based on this, this invention provides a method for generating a multimodal knowledge base. The generated multimodal knowledge base serves as the knowledge reserve for a large language model, improving the accuracy and reliability of the subsequent response content of the large language model and reducing the probability of AI illusions. Furthermore, multimodal information can be used as response content, enriching the diversity of response content and improving user experience.

[0112] Figure 1 This is a flowchart illustrating a multimodal knowledge base generation method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes:

[0113] S11, Obtain the target file;

[0114] S12, parse the target file to obtain the parsing result; the parsing result includes the text content and multimodal information in the target file;

[0115] S13, determine the multimodal labels of the multimodal information, and fuse the text content with the multimodal labels to obtain the fusion result;

[0116] S14. The fusion result is split into knowledge segments to form at least one knowledge card, and a multimodal knowledge base is generated based on at least one knowledge card.

[0117] Specifically, the multimodal knowledge base generation method provided in this embodiment of the invention is executed by a multimodal knowledge base generation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0118] First, execute step S11 to obtain the target file. The target file can be a file uploaded by the first user to build a multimodal knowledge base for generating knowledge cards. The first user can register and log in to the multimodal knowledge base management system via computer. The management system assigns an account to the first user during registration, and this account is bound to at least one multimodal knowledge base. The first user can use this account to log in to the management system and manage the at least one multimodal knowledge base bound to that account, such as creating new multimodal knowledge bases and deleting existing ones. The first user can also manage the knowledge cards in each multimodal knowledge base, such as creating new knowledge cards, deleting existing knowledge cards, and editing the content of existing knowledge cards. In other words, the first user can be the administrator of both knowledge cards and multimodal knowledge bases.

[0119] The content of the target file can be data related to digital human interaction. Accordingly, the subsequently generated multimodal knowledge base can be used for the retrieval of response content from the large language model during the digital human interaction process.

[0120] The target file may include one or more, and each target file may contain text content and multimodal information. The file type of each target file can be any one of the following: txt, pdf, doc, ppt, md (markdown), csv, xls, etc.

[0121] The text content is plain text, while multimodal information may include images, image links, video links, audio links, and Uniform Resource Locator (URL) addresses.

[0122] Then, step S12 is executed to parse the target file and obtain the parsing results. Since the target file contains text content and multimodal information, the text content can be extracted using different methods depending on the file type of the target file.

[0123] When extracting text content, if the target file is a txt, pdf, or doc file, the plain text is extracted, and paragraphs are retained. If the target file is a ppt file, the plain text is extracted, paragraphs are retained, and different text boxes on the same ppt page are separated by line breaks. If the target file is an .md file, no formatting is performed. If the target file is a .csv or .xls file, multimedia content is removed, and only plain text is retained.

[0124] Multimodal information can be obtained directly by parsing the content of the target file. After the multimodal information is determined, it can be stored.

[0125] Next, step S13 is executed to determine the multimodal label of the multimodal information. The multimodal label is used to identify the multimodal information; for example, an image can be identified as...<image_1> ,<image_2> Audio can be identified as...<sound_1> ,<sound_2> Videos can be tagged as...<video_1> ,<video_2> wait.

[0126] Furthermore, to fuse text content with multimodal tags, we first determine the layout position of the multimodal information in the target file, and then insert the corresponding multimodal tags into the corresponding layout positions to obtain the fusion result. This fusion result includes both text content and multimodal tags, but it does not contain the multimodal information itself; instead, it replaces the multimodal information with the corresponding multimodal tags placed in the layout position of the multimodal information.

[0127] Finally, step S14 is executed to perform knowledge decomposition on the fusion result. This can be done by using delimiters such as periods, semicolons, and line breaks in the fusion result, combined with semantic coherence detection technology, to identify the boundaries of knowledge units in the fusion result. Each knowledge unit in the fusion result can constitute a knowledge card, and the fusion result can include one or more knowledge cards. Each knowledge card can include text content and multimodal tags.

[0128] Semantic coherence detection can obtain the embedding vectors of each sentence in the fusion result through the BERT model, and calculate the similarity of the embedding vectors of each sentence to determine the semantically similar sentences.

[0129] The text content in each knowledge card consists of one or more sentences with similar semantics.

[0130] Finally, the knowledge cards are stored to obtain a multimodal knowledge base. This multimodal knowledge base consists of knowledge cards, each of which can be assigned a different number to distinguish and identify the knowledge cards.

[0131] This multimodal knowledge base can be used as a knowledge reserve for large language models, providing a knowledge retrieval library for subsequent interactions between large language models and users, thereby improving the accuracy and reliability of response content during interactions.

[0132] The multimodal knowledge base generation method provided in this embodiment of the invention first obtains a target file; then, it parses the target file to obtain a parsing result, which includes the text content and multimodal information in the target file; subsequently, it determines the multimodal tags of the multimodal information and fuses the text content with the multimodal tags to obtain a fusion result; finally, it performs knowledge decomposition on the fusion result to form at least one knowledge card, and uses the knowledge cards to construct a multimodal knowledge base. This method parses the text content and multimodal information in the target file, fuses the text content with the multimodal tags of the multimodal information, obtains knowledge cards through knowledge decomposition, and then generates a multimodal knowledge base. This decomposes the knowledge in the target file, facilitating subsequent knowledge retrieval, providing knowledge reserves for large language models, improving the accuracy and reliability of the response content provided by the large language model, and reducing the probability of AI illusions. Furthermore, since the knowledge cards contain multimodal tags, response content containing multimodal information can be generated, thereby enriching the diversity of response content and improving the user's interactive experience.

[0133] Based on the above embodiments, the target file is parsed to obtain the parsing result, followed by:

[0134] Determine the access address for multimodal information;

[0135] A mapping table for multimodal information is constructed based on multimodal tags and access addresses.

[0136] Specifically, after parsing the target file, the access address for the multimodal information can be determined first. The multimodal information is then stored, and its storage address is determined, serving as the access address for the multimodal information. It is understood that this storage address can be in the form of a link, a URL, or a file system path, etc., without specific limitations here.

[0137] Subsequently, a mapping table for multimodal information can be constructed using multimodal tags and access addresses. This mapping table can represent the mapping relationship between multimodal tags and access addresses of multimodal information. As shown in Table 1, this mapping table includes not only the multimodal tags and access addresses of multimodal information, but also the index of the multimodal tags.

[0138] Table 1 Mapping Table 1

[0139]

[0140] In this embodiment of the invention, by determining the access address of multimodal information and constructing a mapping table between multimodal tags and access addresses, the access address of multimodal information can be quickly determined based on the multimodal tags, and then the corresponding multimodal information can be obtained by accessing the access address, thus ensuring the information integrity of the knowledge card.

[0141] Based on the above embodiments, in order to facilitate the first user who uploads the target file to confirm whether the content of the generated knowledge card is correct, an access interface can also be configured. The first user can access the multimodal knowledge base bound to the account through the access interface, and then browse the various knowledge cards in the multimodal knowledge base through the access interface.

[0142] Based on the above embodiments, a multimodal knowledge base is generated based on at least one knowledge card, followed by:

[0143] If a knowledge base access request is received, the multimodal label in the first knowledge card corresponding to the knowledge base access request is replaced with the corresponding access address based on the mapping table to obtain the access result, and the access result is displayed on the access interface.

[0144] Specifically, after the multimodal knowledge base is generated, if a first user accesses the multimodal knowledge base through an access interface, the multimodal knowledge base generation device can receive a knowledge base access request. This knowledge base access request corresponds to a first knowledge card, which is the knowledge card that the first user needs to access. It can be any one or more knowledge cards in the multimodal knowledge base, or it can be all the knowledge cards in the multimodal knowledge base.

[0145] The knowledge base access request can be triggered by the first user entering the multimodal knowledge base, at which point the first knowledge card is all the knowledge cards in the multimodal knowledge base. Alternatively, the request can be triggered by the first user selecting the desired first knowledge card after entering the multimodal knowledge base.

[0146] Subsequently, since each knowledge card contains multimodal tags, to ensure the completeness and validity of the displayed content, the multimodal tags in the first knowledge card corresponding to the knowledge base access request can be replaced with the corresponding access address according to the multimodal information mapping table, thus obtaining the access result. This access result is a knowledge card containing both text content and the access address of multimodal information.

[0147] Finally, multimodal information is obtained through the access address, and the access address is replaced to display the access results on the access interface. At this point, the access results displayed on the access interface can be either knowledge cards containing text content and multimodal information, or knowledge cards containing text content and access addresses containing multimodal information. In other words, during the display process, the access address containing multimodal information can either be directly displayed on the access interface or replaced with multimodal information for display; this can be configured according to display requirements, and no specific limitations are made here.

[0148] In this embodiment of the invention, an access scheme for a multimodal knowledge base is provided. When accessing knowledge cards in the multimodal knowledge base, the multimodal tags need to be replaced with the corresponding access addresses so that knowledge cards containing text content and multimodal information or access addresses of multimodal information can be displayed, thereby improving display efficiency and the completeness of the displayed content.

[0149] Based on the above embodiments, a multimodal knowledge base is generated based on at least one knowledge card, and then the following is further included:

[0150] Receive edit requests for the second knowledge card in the multimodal knowledge base;

[0151] Edit the second knowledge card based on the edit request.

[0152] Specifically, after generating the multimodal knowledge base, the first user who uploaded the target file can also edit the knowledge cards. If the first user needs to edit a second knowledge card in the multimodal knowledge base, they can trigger an edit request by clicking the edit button on the editing interface of the second knowledge card. This edit request is used to instruct the user to edit the second knowledge card. The second knowledge card can be any knowledge card in the multimodal knowledge base.

[0153] Afterward, the second knowledge card can be edited according to the editing request. Editing actions can be used to add new multimodal information (such as images, videos, image links, video links, audio links, URLs, etc.) to the second knowledge card, delete existing multimodal information in the second knowledge card, and add and delete plain text content in the second knowledge card.

[0154] In this embodiment of the invention, a function for editing knowledge cards in a multimodal knowledge base is provided to the second user. This not only allows the second user to edit the knowledge cards in the multimodal knowledge base in a timely manner according to their own personalized needs, but also allows the second user to promptly discover and correct errors in the knowledge cards in the multimodal knowledge base.

[0155] Based on the above embodiments, the second knowledge card is edited according to an editing request, including:

[0156] If the editing request is to add target multimodal information, then determine the target multimodal tag and target access address of the target multimodal information;

[0157] Store the target multimodal label and target access address into the mapping table, and add the target multimodal label to the second knowledge card.

[0158] Specifically, in the case where a second user needs to add target multimodal information to a second knowledge card, the target multimodal information to be added can be stored first to determine the target access address of the target multimodal information, and then the target multimodal tag corresponding to the target multimodal information can be generated.

[0159] Subsequently, the target access address and target multimodal tag are stored in the mapping table, and the target multimodal tag is added to the layout position of the target multimodal information in the second knowledge card.

[0160] In this embodiment of the invention, when adding multimodal information to a knowledge card, the mapping table needs to be updated synchronously to ensure that all multimodal information contained therein is determined in a timely manner when the knowledge card is displayed or applied.

[0161] Based on the above embodiments, the text content includes content fragments, which are determined based on the file type of the target file.

[0162] Specifically, the text content of the target file may include one or more content fragments, and the length and number of each content fragment in the target file can be determined by the file type of the target file. In this case, the fusion result includes one or more content fragments, and the layout position of the multimodal information in at least one content fragment corresponds to a multimodal tag.

[0163] Here, the target file type can include text files, presentation files, lightweight text files, and data files. Text files can be of type txt, pdf, doc, etc., presentation files can be of type ppt, lightweight text files can be of type md, and data files can be of type csv, xls, etc.

[0164] In this embodiment of the invention, by extracting content fragments from the target file, the processing efficiency of target files of different file types can be improved, thereby improving the accuracy and precision of subsequent knowledge decomposition.

[0165] Based on the above embodiments, a mapping table for multimodal information is constructed based on multimodal tags and access addresses, including:

[0166] Extracting summaries of multimodal information;

[0167] A mapping table is constructed based on multimodal tags, access addresses, and summaries.

[0168] Specifically, to prevent similar multimodal tags from being too close together in the multimodal knowledge base, which could lead to incorrect referencing of knowledge cards in the subsequent large language model, the mapping table for multimodal information can also include a summary of the multimodal information. This increases the distinguishability between the corresponding content of each multimodal information in the mapping table. Here, the summary of the multimodal information can be obtained by parsing and extracting the multimodal information, and can include the title and brief description of the multimodal information. For example, the mapping table for multimodal information can be shown in Table 2:

[0169] Table 2 Mapping Table 2

[0170]

[0171] In this embodiment of the invention, a summary of the multimodal information can also be introduced into the mapping table of multimodal information to increase the distinguishability between the corresponding contents of each multimodal information in the mapping table, thereby helping the subsequent large language model to correctly reference the knowledge cards in the multimodal knowledge base.

[0172] Based on the above embodiments, the steps for determining content fragments include:

[0173] In response to a file of type text, the text content of the text file is treated as a content fragment;

[0174] In response to a file type of demo file, the text content of the demo file is divided into one or more content fragments based on the page content of the demo file;

[0175] In response to a file type of lightweight text file, the text content of the lightweight text file is divided into one or more content segments according to the minimum heading of the lightweight text file;

[0176] In response to a file type of data file, the text content of the data file is divided into one or more content fragments based on the layout format of the data file.

[0177] Specifically, when determining the content fragments of a target file, if the target file is a text file, the text content of the text file can be considered as a content fragment. That is, the text content extracted from each target file of type txt, pdf, doc, etc., can be merged into a single content fragment.

[0178] When the target file type is a presentation file, the text content of the presentation file can be divided into one or more content fragments according to the page content of the presentation file. That is, the text content extracted from each target file of type PPT is divided into different content fragments according to the PPT page, and each page of the PPT is a content fragment. Each content fragment stores data in the form of key-value pairs. Each content fragment contains indexes (i.e., keys) such as PPT name, current page number, and PPT details, as well as corresponding data (i.e., values).

[0179] like Figure 2 The image shown is a schematic diagram of the content on one of the slides in a PowerPoint presentation. Figure 2 A content fragment obtained can be represented as:

[0180] PPT Title: Educational Applications in the Large Model Industry

[0181] Current page number: Page 7

[0182] Details:

[0183] A New Operating System for the Era of Large Models: Doubao Large Model Refined with Hundreds of Billions of Tokens Daily

[0184] Internally, over 50 business units use the Doubao big model, covering various scenarios such as collaborative office work, data analysis, copywriting, assisted programming, content moderation, customer service, game NPCs, role-playing dialogue, and education.

[0185] 01 Content moderation business

[0186] The absolute value of classification accuracy improved by 8% (86% → 94%).

[0187] 02 A certain search relevance task

[0188] Accuracy improved by 25% compared to online BERT models.

[0189] 03 A marketing copy generation scenario

[0190] The approval rate for GPT4 manual review is 92%, while the approval rate for Doubao SFT is 5%.

[0191] When the target file is a lightweight text file, the text content of the lightweight text file can be divided into one or more content segments according to the minimum heading of the lightweight text file. That is, the text content extracted from each target file of type md will be divided into different content segments according to the minimum heading.

[0192] When the target file is a data file, its text content can be segmented into one or more content fragments based on the data file's layout format. This means dividing the extracted text content into different fragments according to the layout format of each target file type, such as CSV or XLS. Here, the layout format can include tables with and without header rows. For example, each row in the data file except for the header row can be treated as a content fragment, with each column separated by a newline character. The specific content of each fragment is adjusted based on whether a header row exists. If a header row exists, each column contains the column name from the header row; otherwise, the column name is not included.

[0193] In this embodiment of the invention, a scheme for determining content fragments is provided for target files of different file types, which can ensure the content division of target files with diverse types.

[0194] Based on the above embodiments, the data file is specifically a table file;

[0195] Based on the layout format of data files, the text content of data files is segmented into one or more content fragments, including:

[0196] If the first row of the table file is the header row, then each row in the table file other than the first row is treated as a content fragment, and the column content in each content fragment includes the column headers in the header row.

[0197] If the first row of the table file is not the header row, then each row in the table file will be treated as a content segment.

[0198] Specifically, when the data file is a table file, it can be determined whether the first row of the table file is a header row, and the file content can be divided into one or more content segments based on the determination result. If the first row of the table file is a header row, then each row in the table file except the first row can be treated as a content segment, and the column content in each content segment can be determined to include the column headers in the header row; if the first row of the table file is not a header row, then each row in the table file can be treated as a content segment.

[0199] At the same time, it is necessary to ensure that each column of content is separated by a newline character.

[0200] Table 3 shows a schematic diagram of a table file where the first row is the header row.

[0201] Table 3, Table File 1

[0202]

[0203] The table file in Table 3 can be split into four content segments as shown in Table 4:

[0204] Table 4 Content Segment Table 1

[0205]

[0206] Table 5 shows a schematic diagram of a table file where the first row is not the header row.

[0207] Table 5, Table Document 2

[0208]

[0209] The table file in Table 5 can be split into four content segments as shown in Table 6:

[0210] Table 6 Content Segment Table 2

[0211]

[0212] In this embodiment of the invention, when the data file is specifically a table file, the table file can be divided into one or more content segments based on whether the first row of the table file is a header row. This can ensure the integrity of the content segments and avoid leaving information in the table file.

[0213] Based on the above embodiments, the fusion result is decomposed into knowledge cards to form at least one knowledge card, including:

[0214] Determine the split delimiter and knowledge card length threshold;

[0215] Based on the splitting delimiter and the knowledge card length threshold, the fusion result is split into a first set of knowledge fragments;

[0216] Based on the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold, at least one knowledge card is obtained by applying the first knowledge fragment set.

[0217] Specifically, in this embodiment of the invention, when performing knowledge segmentation on the fusion result of text content and multimodal tags, a segmentation separator and a knowledge card length threshold can be determined first. The segmentation separator is the basis for segmenting knowledge fragments and is used to segment knowledge fragments. The knowledge card length threshold can be used as a criterion for determining whether a knowledge fragment can be used as a knowledge card, and can be expressed in terms of word count to characterize the length requirement of the knowledge card. The segmentation separator and the knowledge card length threshold can be preset or set by the first user, and no specific limitation is made here.

[0218] If both the splitting delimiter and the knowledge card length threshold are preset, an automatic splitting strategy is used to split the fusion result into knowledge segments. The preset splitting delimiter can be a newline character, and the preset knowledge card length threshold can be 512 characters.

[0219] If both the delimiter and the knowledge card length threshold are set by the first user, a rule-based delimiter strategy is used to split the fusion result into knowledge segments. The delimiter set by the first user can be a common single character, such as a comma (,), semicolon (;), tab (\t), pipe (|), space, or newline (\n), or other special characters, such as colon (:), forward slash ( / ), backslash (\), #, $, %, &, *, etc. Additionally, it can be a blank sequence composed of one or more spaces, tabs, or newlines. The knowledge card length threshold set by the first user can be 512, 1024, or other similar numbers of characters.

[0220] Furthermore, the fusion result can be divided into a first set of knowledge fragments using splitting delimiters and knowledge card length thresholds. Since the fusion result includes one or more content fragments, the layout position of multimodal information in each content fragment corresponds to a multimodal tag. Therefore, knowledge splitting of the fusion result can be understood as splitting each content fragment in the fusion result. First, based on the knowledge card length threshold, the content fragments to be split in the fusion result (i.e., content fragments whose length exceeds the knowledge card length threshold) can be determined, and the content between two adjacent splitting delimiters in the content fragment to be split can be considered as one knowledge fragment.

[0221] For content fragments in the fusion result that do not need to be split, they can be directly regarded as a knowledge fragment. All knowledge fragments in the fusion result constitute the first knowledge fragment set.

[0222] The length of each knowledge fragment in the first knowledge fragment set can also be represented by the number of characters. Using the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold, knowledge fragments in the first knowledge fragment set whose length is greater than the knowledge card length threshold can be segmented so that the length of the segmented knowledge fragments is less than or equal to the knowledge card length threshold. At this time, each segmented knowledge fragment can be used as a knowledge card. In addition, the standard of length less than or equal to the knowledge card length threshold can be used to splice at least two adjacent knowledge fragments in the first knowledge fragment set whose length is less than the knowledge card length threshold. Each spliced ​​knowledge fragment or segmented knowledge fragment can be used as a knowledge card.

[0223] Understandably, to ensure the semantic integrity of knowledge fragments, they need to be segmented by sentence as the smallest unit, meaning that each sentence in the first set of knowledge fragments cannot be broken down.

[0224] In this embodiment of the invention, the fusion result is split by splitting delimiters, and at least one knowledge card in the first knowledge fragment set is determined by combining the knowledge card length threshold and the length of each knowledge fragment in the first knowledge fragment set. This can make the length of each knowledge card moderate, which avoids the situation that a single knowledge card has too much content and also ensures the semantic integrity of each knowledge card.

[0225] Based on the above embodiments, and based on the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold, the first knowledge fragment set is applied to obtain at least one knowledge card, including:

[0226] For any knowledge fragment in the first set of knowledge fragments, if the length of any knowledge fragment does not exceed the knowledge card length threshold, then any knowledge fragment is treated as a knowledge card.

[0227] If the length of any knowledge fragment exceeds the knowledge card length threshold, then the knowledge fragment is split into multiple first knowledge segments whose length does not exceed the knowledge card length threshold, and each first knowledge segment is treated as a knowledge card.

[0228] Specifically, in this embodiment of the invention, for any knowledge fragment in the first set of knowledge fragments, the length of the knowledge fragment can be compared with the length threshold of the knowledge card. If the length of the knowledge fragment is less than or equal to the length threshold of the knowledge card, then the length of the knowledge fragment is considered to meet the length requirement of the knowledge card, and the knowledge fragment can be directly used as a knowledge card.

[0229] If the length of any knowledge fragment exceeds the knowledge card length threshold, then the length of any knowledge fragment is considered to exceed the knowledge card length requirement. Therefore, any knowledge fragment needs to be split into multiple first knowledge segments whose length does not exceed the knowledge card length threshold. Each first knowledge segment meets the knowledge card length requirement, so each first knowledge segment can be used as a knowledge card.

[0230] In this embodiment of the invention, a scheme for determining knowledge cards using a knowledge card length threshold is provided, which can ensure that the obtained knowledge cards meet the length requirements.

[0231] Based on the above embodiments, any knowledge fragment is split into multiple first knowledge segments whose length does not exceed the knowledge card length threshold, including:

[0232] Break any piece of knowledge into multiple statements;

[0233] Based on the length of each statement and the knowledge card length threshold, multiple statement sets are determined from multiple statements in the order of statements, and each statement set is treated as a knowledge card;

[0234] The length of each set of statements does not exceed the knowledge card length threshold.

[0235] Specifically, when splitting any knowledge fragment whose length exceeds the knowledge card length threshold, this fragment can be first divided into multiple statements. That is, statements are used as the smallest unit to break down any knowledge fragment into multiple statements. Each statement is a complete sentence that can represent complete semantics.

[0236] Subsequently, using the length of each statement and the knowledge card length threshold, multiple statement sets are determined from multiple statements in order. Each statement obtained from the splitting process is traversed. For the current statement, if its length does not exceed the knowledge card length threshold, it is further checked whether the current statement and its adjacent next statement together exceed the knowledge card length threshold. If not, the current statement and the next statement are merged into a statement set as the new current statement. This process continues until the new current statement and its adjacent next statement together exceed the knowledge card length threshold, at which point the new current statement is treated as a knowledge card.

[0237] If the length of the current statement exceeds the knowledge card length threshold, the current statement can be directly used as a knowledge card.

[0238] Therefore, each statement set can include one or more statements. If the length of the current statement exceeds the knowledge card length threshold, the current statement constitutes a statement set on its own. If the length of the current statement does not exceed the knowledge card length threshold, the current statement and one or more adjacent next statements together constitute a statement set.

[0239] Since the knowledge card length threshold is used as the criterion for determining the set of statements, the length of each set of statements does not exceed the knowledge card length threshold.

[0240] In this embodiment of the invention, by splitting knowledge fragments into multiple statements and determining knowledge cards in combination with a knowledge card length threshold, it can be ensured that the length of the knowledge card does not exceed the knowledge card length threshold and that the semantics are complete.

[0241] Based on the above embodiments, the fusion result is decomposed into knowledge cards to form at least one knowledge card, and the method further includes:

[0242] Determine the knowledge redundancy length threshold, and based on the current knowledge card, apply the knowledge redundancy length threshold to identify redundant knowledge;

[0243] Copy redundant knowledge to the starting position of the next knowledge card after the current knowledge card.

[0244] Specifically, since both automatic and rule-based splitting strategies are used to split each content fragment in the fusion result, these are considered hard splitting and do not take into account the semantic relationships between statements. Therefore, to prevent the content of a single knowledge point from being split into different knowledge cards, resulting in incomplete knowledge card content, this embodiment of the invention introduces a knowledge redundancy mechanism.

[0245] First, determine the knowledge redundancy length threshold. Under the automatic splitting strategy, this threshold can be preset; under the rule-based splitting strategy, it can be set by the first user. Here, the redundancy length threshold can be represented by the number of characters, such as 100 characters, or other character counts; no specific limitation is made here.

[0246] Then, from each knowledge card obtained by knowledge decomposition (excluding the last knowledge card), a combination of one or more statements whose last value does not exceed the knowledge redundancy length threshold can be selected as redundant knowledge.

[0247] Finally, the redundant knowledge is copied and added to the beginning of the next knowledge card. In other words, in two adjacent knowledge cards, the last part of the previous knowledge card is identical to the beginning part of the next knowledge card; the length of this identical content is the length of the redundant knowledge.

[0248] In this embodiment of the invention, a knowledge redundancy mechanism is introduced to avoid the last part of a knowledge card being forcibly split into a subsequent knowledge card due to the length limit of the knowledge card, which would result in incomplete content of the knowledge card and thus ensure the knowledge integrity of the knowledge card.

[0249] Based on the above embodiments, and using the current knowledge card, a knowledge redundancy length threshold is applied to determine redundant knowledge, including:

[0250] Determine whether there is a second knowledge segment at the end of the current knowledge card that does not exceed the knowledge redundancy length threshold;

[0251] If there is no second knowledge segment at the end of the current knowledge card, then the last statement of the current knowledge card will be regarded as redundant knowledge.

[0252] If a second knowledge segment exists at the end of the current knowledge card, redundant knowledge is determined based on the length of the second knowledge segment and the length of the next knowledge card after the current knowledge card.

[0253] Specifically, when identifying redundant knowledge, we can first determine whether there is a second knowledge segment at the end of the current knowledge card that does not exceed the knowledge redundancy length threshold. The second knowledge segment can consist of one or more statements whose length does not exceed the knowledge redundancy length threshold, and the length of the second knowledge segment does not exceed the knowledge redundancy length threshold.

[0254] When determining the second knowledge segment, start from the last statement of the current knowledge card in reverse order and check if the length of each statement exceeds the knowledge redundancy length threshold. If the last statement is shorter than the knowledge redundancy length threshold, check if the length of the combination of the last statement and the second-to-last statement exceeds the knowledge redundancy length threshold, and so on, until the length of the combined statement exceeds the knowledge redundancy length threshold. At this point, the combination of statements can be used as the second knowledge segment.

[0255] Next, determine whether there is a second knowledge segment at the end of the current knowledge card. If there is, it means that the end of the current knowledge card contains at least one statement whose length does not exceed the knowledge redundancy length threshold. At this time, redundant knowledge can be determined based on the length of the second knowledge segment and the length of the next knowledge card after the current knowledge card, so that the length of the next knowledge card will not exceed the knowledge redundancy length threshold after adding redundant knowledge.

[0256] If there is no second knowledge fragment at the end of the current knowledge card, it means that the length of the last statement of the current knowledge card exceeds the knowledge redundancy length threshold, and the last statement of the current knowledge card can be directly regarded as redundant knowledge.

[0257] Therefore, when identifying redundant knowledge, there is a special case where the length of the last statement of the current knowledge card exceeds the knowledge redundancy length threshold, but it is still regarded as redundant knowledge. The purpose is to ensure the semantic integrity of the redundant knowledge.

[0258] In addition, another special case is that although the length of the last statement of the current knowledge card does not exceed the knowledge redundancy length threshold, if it is added to the next knowledge card, the length of the next knowledge card will exceed the knowledge card length threshold. In this case, the last statement of the current knowledge card needs to be regarded as redundant knowledge.

[0259] In this embodiment of the invention, redundant knowledge is determined by judging whether there is a second knowledge fragment at the end of the knowledge card. This can make redundant knowledge as concise as possible and avoid a large amount of duplicate content in adjacent knowledge cards.

[0260] Based on the above embodiments, the present invention provides complete implementation methods for automatic splitting strategy and rule-based splitting strategy respectively.

[0261] For the automatic splitting strategy, the newline character is used as the splitting delimiter to split each content fragment in the fusion result into knowledge fragments, and each knowledge fragment is a paragraph.

[0262] First, the word count of the content fragment is checked based on the knowledge card length threshold. If the length of the content fragment exceeds the knowledge card length threshold, the content fragment is split into paragraphs, and then the paragraphs are merged or split.

[0263] Specifically, the paragraphs are processed sequentially from front to back. If the length of the current paragraph does not exceed the knowledge card length threshold, it is determined whether the overall length of the current paragraph and the next paragraph exceeds the knowledge card length threshold. If not, the current paragraph and the next paragraph are merged into a new current paragraph. This process continues until the overall length of the current paragraph and the next paragraph exceeds the knowledge card length threshold. In this case, the current paragraph is treated as a knowledge fragment, which is then used as a knowledge card.

[0264] If the length of the current paragraph exceeds the knowledge card length threshold, the current paragraph can be split according to the default settings (such as according to punctuation marks), splitting the current paragraph into one or more statements, and then merging the statements in the order of the statements from front to back.

[0265] Furthermore, if the length of the current statement does not exceed the knowledge card length threshold, then it is determined whether the overall length of the current statement and the next statement exceeds the knowledge card length threshold. If it does not exceed the threshold, the current statement and the next statement are merged into a new current statement. This process continues until the overall length of the new current statement and the next statement exceeds the knowledge card length threshold. In this case, the current statement is identified as a statement set and treated as a knowledge card.

[0266] If the length of the current statement exceeds the knowledge card length threshold, the current statement can be directly identified as a statement set and treated as a knowledge card.

[0267] It should be noted that during the knowledge decomposition process, the last part of the previous knowledge card is copied and added to the beginning of the next knowledge card as redundant knowledge.

[0268] If the length of the last statement in the previous knowledge card exceeds the knowledge card length threshold, it still needs to be added as redundant knowledge to the beginning of the next knowledge card.

[0269] If the length of the last statement in the preceding knowledge card does not exceed the knowledge card length threshold, but the sum of the lengths of the last statement and the next knowledge card exceeds the knowledge card length threshold, it still needs to be added as redundant knowledge to the beginning of the next knowledge card.

[0270] For the rule splitting strategy, firstly, the word count of each content fragment in the fusion result is checked according to the knowledge card length threshold. If the length of the content fragment exceeds the knowledge card length threshold, the content fragment is split according to the splitting delimiter to obtain multiple knowledge fragments. The knowledge fragments are combined to obtain the first fragment set, and then processed from front to back according to the order of each knowledge fragment.

[0271] If the length of any knowledge fragment does not exceed the knowledge card length threshold, then that knowledge fragment is directly treated as a knowledge card.

[0272] If the length of any knowledge fragment exceeds the knowledge card length threshold, then the knowledge fragment will be further split into multiple statements according to the default settings (such as according to punctuation marks), and then the statements will be merged from front to back in the order of the statements.

[0273] If the length of the current statement does not exceed the knowledge card length threshold, then determine whether the overall length of the current statement and the next statement exceeds the knowledge card length threshold. If not, merge the current statement and the next statement as a new current statement, until the overall length of the current statement and the next statement exceeds the knowledge card length threshold. Then, determine the current statement as a statement set and use it as a knowledge card.

[0274] If the length of the current statement exceeds the knowledge card length threshold, the current statement can be directly identified as a statement set and treated as a knowledge card.

[0275] It should be noted that during the knowledge decomposition process, the last part of the previous knowledge card is copied and added to the beginning of the next knowledge card as redundant knowledge.

[0276] If the length of the last statement in the previous knowledge card exceeds the knowledge card length threshold, it still needs to be added as redundant knowledge to the beginning of the next knowledge card.

[0277] If the length of the last statement in the preceding knowledge card does not exceed the knowledge card length threshold, but the sum of the lengths of the last statement and the next knowledge card exceeds the knowledge card length threshold, it still needs to be added as redundant knowledge to the beginning of the next knowledge card.

[0278] Based on the above embodiments, the fusion result is decomposed into knowledge cards to form at least one knowledge card, and the method further includes:

[0279] Extract title information and topic transition points from the fusion results;

[0280] Based on the title information and topic shift points, the fusion result is split into a second set of knowledge fragments;

[0281] Based on the title information and the second set of knowledge fragments, at least one knowledge card is determined.

[0282] Specifically, when performing knowledge decomposition on the fusion results, in addition to using automatic decomposition strategies and rule-based decomposition strategies, model-based decomposition strategies can also be used.

[0283] The model splitting strategy refers to leveraging the powerful capabilities of large language models during the knowledge splitting process of the fusion results to make the resulting knowledge cards more accurate and reliable.

[0284] When implementing a model splitting strategy, you can first use a large language model to extract title information and topic transition points from the fusion results.

[0285] Title information extraction can be achieved by first determining the paragraphs in the target file based on its file type. For example, for target files of type txt, doc, and pdf, a single paragraph can be considered as a paragraph; for target files of type ppt, a single page can be considered as a paragraph.

[0286] Next, each paragraph in the target file is numbered, with each paragraph having a corresponding paragraph number. Each paragraph in the target file is then input into a large language model, which outputs the title information of the target file. This title information can include various levels of headings, their corresponding levels, and paragraph numbers, and can be represented using the title tree of the target file.

[0287] Topic shift points can be extracted using algorithms such as Recursive Chunking, Small2Big, Semantic Chunking, and LumberChunker. Here, a topic shift point is the paragraph location in the target file where a topic shift occurs. The target file may contain one or more topic shift points, and the content between two adjacent topic shift points constitutes a content topic block in the target file.

[0288] Taking the LumberChunker algorithm as an example, the initial topic transition point is taken as the starting position of the target file. This initial topic transition point is then used as the current topic transition point. The next topic transition point is determined by the large language model, taking the continuous paragraph content following the current topic transition point that does not exceed the knowledge card length threshold. This process continues until the end of the target file is reached.

[0289] Subsequently, using title information and topic shift points, the fusion result is split into a second set of knowledge fragments. For example, each content segment in the fusion result can be split according to the title information to obtain an initial split result. Then, using topic shift points, the longer content in the initial split result can be split a second time to obtain multiple knowledge fragments that constitute the second set of knowledge fragments.

[0290] Finally, combining the title information and the second set of knowledge fragments, one or more knowledge cards are determined. Specifically, corresponding title information can be added to the knowledge fragments in the second set that correspond to the title information to obtain each knowledge card. Therefore, the number of knowledge cards is equal to the number of knowledge fragments in the second set. The only difference between knowledge cards and knowledge fragments in the second set is that knowledge cards may contain title information.

[0291] In this embodiment of the invention, by utilizing title information and topic transition points, and by using a large language model to perform knowledge decomposition, the topic integrity and topic independence of the obtained knowledge cards can be guaranteed.

[0292] Based on the above embodiments, the fusion result is divided into a second set of knowledge fragments based on title information and topic transition points, including:

[0293] Based on the title information, determine the initial set of knowledge fragments for the fusion result;

[0294] If it is determined that there is a specified knowledge fragment in the initial knowledge fragment set whose length exceeds the first preset length, and there is a topic transfer point in the specified knowledge fragment, then the specified knowledge fragment is segmented to obtain the segmentation result. The non-specified knowledge fragments in the initial knowledge fragment set and the segmentation result are merged to obtain an intermediate knowledge fragment set.

[0295] If it is determined that there are two adjacent knowledge fragments with the same title level in the intermediate knowledge fragment set, and the length of at least one of the two adjacent knowledge fragments is less than the second preset length, then the two adjacent knowledge fragments are merged to obtain the second knowledge fragment set.

[0296] Specifically, in the process of splitting the fusion result into a second set of knowledge fragments, the title information can be used to determine the initial set of knowledge fragments of the fusion result. That is, the position of each level of title in the title information is used as the dividing point to split each content fragment in the fusion result, resulting in multiple knowledge fragments constituting the initial set of knowledge fragments.

[0297] Next, it is determined whether there is a specified knowledge fragment in the initial knowledge fragment set whose length exceeds the first preset length. The first preset length can be set as needed, and can be the same as or different from the knowledge card length threshold. No specific limitation is made here.

[0298] If the initial set of knowledge fragments contains a specified knowledge fragment exceeding a first preset length, the system continues to determine if a topic transition point exists within that specified knowledge fragment. If a topic transition point exists within the specified knowledge fragment, it is considered that the specified knowledge fragment contains at least two topics. To ensure that each knowledge fragment has one topic, the specified knowledge fragment can be segmented to obtain a segmentation result. This segmentation result contains multiple knowledge fragments resulting from the segmentation of all specified knowledge fragments.

[0299] Subsequently, the non-specified knowledge fragments and the segmentation results in the initial knowledge fragment set are merged to obtain an intermediate knowledge fragment set. Here, non-specified knowledge fragments refer to knowledge fragments whose length does not exceed a first preset length. In this embodiment of the invention, to prevent resource waste due to insufficient knowledge card content, non-specified knowledge fragments are not split regardless of whether there are topic shift points within them. The length of each knowledge fragment included in this intermediate knowledge fragment set does not exceed the first preset length.

[0300] Finally, it checks whether two adjacent knowledge fragments in the intermediate knowledge fragment set have the same topic level. If two adjacent knowledge fragments with the same topic level exist in the intermediate knowledge fragment set, it further checks whether the length of these two knowledge fragments is less than the second preset length. If at least one of the two adjacent knowledge fragments is shorter than the second preset length, these two knowledge fragments can be merged to obtain the second knowledge fragment set. This allows knowledge fragments with the same title level and shorter length to be merged, preventing the knowledge fragments from being split too finely. The second preset length can be set as needed, for example, it can be set to 30 or other values. Each knowledge fragment in the second knowledge fragment set corresponds to a title level, and different knowledge fragments may correspond to the same or different title levels.

[0301] Furthermore, by utilizing the title information and the second set of knowledge fragments, each knowledge card is determined to correspond to a title branch of the title tree, and the title branch consists of the current title and all the parent titles of that title.

[0302] In this embodiment of the invention, by combining the length of each knowledge fragment in the initial knowledge fragment set and the title tree, a second knowledge fragment set is determined. This ensures that each knowledge card corresponds to a title branch of appropriate length, avoiding excessive or insufficient content in the knowledge cards, which would affect readability.

[0303] Building upon the above embodiments, each user's account corresponds to at least one separate multimodal knowledge base. The knowledge cards generated after each user uploads a target file are stored in the multimodal knowledge base bound to their respective account. Correspondingly, each multimodal knowledge base corresponds to a mapping table of multimodal information. This approach ensures data security and prevents leaks, while also reducing the number of multimodal tags in the mapping table, thus decreasing the likelihood of incorrect references in subsequent large language models.

[0304] Based on the above embodiments, a multimodal knowledge base is generated based on at least one knowledge card, and then the following is further included:

[0305] If a sharing request for a third knowledge card is received, determine the target account corresponding to the sharing request;

[0306] In response to the target account's request to allow sharing, a third knowledge card is generated in the target account's multimodal knowledge base.

[0307] Specifically, after generating the multimodal knowledge base, the first user can share knowledge cards from the constructed multimodal knowledge base with other users. Simultaneously, the first user can also receive knowledge cards shared by other users.

[0308] If a first user needs to share a third knowledge card with a third user, the first user can trigger a sharing request for the third knowledge card by clicking the share button on the sharing interface of the third knowledge card. This sharing request includes the target account and the third knowledge card, where the target account is the sharing subject (i.e., the third user's account), and the third knowledge card is the sharing object. The third knowledge card can be any knowledge card from the first user's multimodal knowledge base.

[0309] Upon receiving a sharing request for a third knowledge card, the multimodal knowledge base generation device can identify the target account corresponding to the sharing request, send authorization information for the sharing request to the target account, and receive feedback information from the target account regarding the authorization information. Here, the authorization information is used to confirm whether the third user agrees to receive the third knowledge card shared by the first user, and the feedback information is the result of whether the third user agrees to receive the third knowledge card. The feedback information can include "agree to receive" and "disagree to receive," where "agree to receive" means the target account allows the sharing request, and "disagree to receive" means the target account allows the sharing request.

[0310] If the target account allows the sharing request, that is, if the third user agrees to receive the third knowledge card, the third knowledge card can be generated in the target account's multimodal knowledge base. In other words, the third knowledge card and its corresponding mapping table are copied to the target account's multimodal knowledge base.

[0311] In this embodiment of the invention, knowledge cards in a multimodal knowledge base can be shared among different users.

[0312] Because existing digital human interaction methods cannot answer questions beyond the scope of a pre-set question bank, nor can they flexibly organize the multimodal information needed based on the user's specific requirements, therefore, based on the above embodiments, such as Figure 3 As shown, this embodiment of the invention also provides a multimodal information retrieval enhancement generation method, including:

[0313] S21, Obtain interactive information;

[0314] S22, extract the semantic features of the interaction information, and based on the semantic features, determine the target knowledge card in the multimodal knowledge base that matches the semantic features of the interaction information;

[0315] S23, based on the target knowledge card, apply the large language model to generate the response content corresponding to the interactive information;

[0316] The multimodal knowledge base is generated based on the multimodal knowledge base generation method described above.

[0317] Specifically, the multimodal information retrieval enhancement generation method provided in this embodiment of the invention is executed by a multimodal information retrieval enhancement generation device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0318] First, step S21 is executed to obtain interaction information. This interaction information may be a question that the fourth user needs to answer through the large language model. This interaction information may be information in the form of voice or text input by the fourth user through the interactive interface.

[0319] Here, the interactive information can be initially answered using knowledge cards in the multimodal knowledge base generated by the multimodal knowledge base generation method provided in the above embodiments; that is, the interactive information has the same semantics as the knowledge cards in the multimodal knowledge base. Therefore, the multimodal knowledge base can provide knowledge reserves for large language models.

[0320] Next, step S22 is executed, where the interaction information is input into the large language model, which can then extract semantic features of the interaction information. These semantic features can be in the form of high-dimensional vectors. It is understandable that, since the interaction information may not meet the retrieval requirements of the large language model, the model can first understand the interaction information and rewrite it, converting it into a content format that meets the retrieval requirements.

[0321] Furthermore, the semantic features of the interactive information can be used to calculate the similarity between the interactive information and each knowledge card in the multimodal knowledge base. This similarity can be the content similarity between the interactive information and each knowledge card, for example, it can be obtained by calculating the semantic similarity between the semantic features of the interactive information and the semantic features of each knowledge card. The semantic similarity between each knowledge card can also be obtained through a large language model.

[0322] Next, a preset number of knowledge cards with high similarity are selected from all knowledge cards as target knowledge cards. The preset number can be set as needed, for example, it can be 3, 5, etc., and there is no specific limitation here.

[0323] Finally, step S23 is executed. Using the target knowledge card obtained in step S23, the powerful generation capabilities of the large language model are applied to generate the response content corresponding to the interactive information. Specifically, the target knowledge card and the interactive information are input into the large language model, which then uses the content of the target knowledge card to generate the response content corresponding to the interactive information. This response content includes text content and may also contain multimodal tags.

[0324] It should be noted that the multimodal tags contained in the response content are generated by a large language model. The large language model may generate incorrect or non-standard multimodal tags. Therefore, this embodiment of the invention also needs to pre-design regular expression judgment rules for multimodal tags to delete incorrect multimodal tags and correct non-standard multimodal tags in order to ensure the correctness and standardization of the multimodal tags contained in the response content.

[0325] The multimodal information retrieval enhancement generation method provided in this embodiment of the invention applies retrieval enhancement technology based on a large language model. It first selects matching target knowledge cards from a multimodal knowledge base, and then uses the large language model to combine the target knowledge cards to obtain response content. The entire interaction process does not require limiting the user's input content, allowing for personalized and diverse interactions while ensuring the authenticity and reliability of the response content, thus improving its quality and meeting users' personalized interaction needs. Furthermore, this method decouples the large language model from the multimodal knowledge base in the system architecture, allowing for further improvement in the quality of the response content generated by the large language model by updating the knowledge cards in the multimodal knowledge base. Deploying different multimodal knowledge bases also enables the large language model to generate response content from different domains, thereby adapting to different application scenarios and effectively reducing the deployment cost of the software system.

[0326] Based on the above embodiments, target knowledge cards that match the semantic features of interactive information in the multimodal knowledge base are determined based on semantic features, including:

[0327] Determine the knowledge vector for each knowledge card in the multimodal knowledge base;

[0328] The target knowledge card is determined based on the similarity between semantic features and knowledge vectors.

[0329] Specifically, when determining the target knowledge card, the knowledge vector of each knowledge card in the multimodal knowledge base can be determined first. The knowledge vector of each knowledge card can be obtained either by processing the knowledge cards using knowledge vectorization techniques after the multimodal knowledge base is generated, or it can be generated in real-time when applying the multimodal knowledge base; no specific limitation is made here. Furthermore, each knowledge vector can correspond to an index.

[0330] As can be seen from the foregoing description, the knowledge cards in the embodiments of the present invention contain text content and multimodal tags. Therefore, when determining the knowledge vector of each knowledge card, if the pre-constructed multimodal information mapping table records the title and brief description of multimodal information, the title and brief description of multimodal information can be queried based on the multimodal tags, and the knowledge vector of the knowledge card can be generated together with the text content in the knowledge card.

[0331] Subsequently, the semantic features of the interactive information can be matched with each knowledge vector, and the similarity between the semantic features of the interactive information and each knowledge vector can be calculated. By sorting the similarity from high to low, the top preset number of target knowledge cards with high similarity can be obtained.

[0332] In this embodiment of the invention, by calculating the similarity between the semantic features of the interactive information and the knowledge vectors of each knowledge card in the multimodal knowledge base, the target knowledge card is determined from each knowledge card. This can improve the accuracy of the target knowledge card and also improve the efficiency of determining the target knowledge card when the knowledge vectors of each knowledge card are predetermined, thereby enhancing the efficiency of multimodal information retrieval and generation.

[0333] Based on the above embodiments, the response content includes target multimodal tags, and the target multimodal tags correspond to target access addresses; based on the target knowledge card, a large language model is applied to generate response content corresponding to the interaction information, which then includes:

[0334] Based on the target access address, determine the target multimodal information in the response content and display the response content on the interactive interface.

[0335] Specifically, since the response content is obtained by combining the target knowledge card and interaction information from the large language model, if the target knowledge card contains a target multimodal tag, then the response content generated based on the content of the target knowledge card may contain the target multimodal tag. In other words, the response content may contain at least one target multimodal tag from the target knowledge card.

[0336] A target multimodal tag can correspond to a target access address, which can be the storage location of the multimodal information corresponding to the target multimodal tag. Therefore, after generating the response content, the target multimodal information in the response content can be determined using the target access address; that is, the multimodal information stored in the target access address is identified as the target multimodal information, and this target multimodal information is included in the response content.

[0337] Subsequently, the multimodal tags in the response content can be replaced with target multimodal information, and the response content can be displayed on the real interface. At this time, the response content displayed on the interface contains text content and target multimodal information.

[0338] In this embodiment of the invention, when the response content contains a target multimodal tag, the response content containing the target multimodal information can be displayed on the interactive interface, thereby achieving multimodal response to the user and improving the user experience.

[0339] Based on the above embodiments, the modal knowledge base generation device and the multimodal information retrieval enhancement generation device are both located in the back end, while the access interface and the interactive interface are both located in the front end.

[0340] Figure 4 This is a schematic diagram illustrating the overall process of the modal knowledge base generation method and the degree modality information retrieval enhancement generation method in an embodiment of the present invention, as shown below. Figure 4 As shown, the process includes user behavior, pre-splitting format processing, knowledge splitting, front-end knowledge card display, multimodal information parsing and saving, and multimodal information acquisition and output.

[0341] During the user's action, the user uploads the target file and selects a splitting strategy, which may include automatic splitting strategy, large model splitting strategy, and rule-based splitting strategy.

[0342] Then, the backend parses the target file, extracts the text content and multimodal information of the target file. The text content undergoes a pre-splitting format processing process, while the multimodal information undergoes a multimodal information acquisition and output process.

[0343] During the parsing and storage of multimodal information, the backend parses the multimodal information, stores it, obtains the access address of the multimodal information, and determines the multimodal tag of the multimodal information. Here, the multimodal tag can be obtained by converting the access address of the multimodal information. The backend uses an algorithm to determine the summary of the multimodal information and, together with the multimodal tag, constructs a mapping table for the multimodal information.

[0344] During the format processing before splitting, the backend combines the multimodal tags with the text content of the target file, adding the multimodal tags to the layout position of the multimodal information in the target file, resulting in text content containing multimodal tags, which is the fusion result.

[0345] Knowledge is split using one of the following splitting strategies: user-selected automatic splitting strategy, large model splitting strategy, and rule-based splitting strategy, resulting in knowledge cards with multimodal labels.

[0346] When a user needs to add new multimodal information, the front end displays existing knowledge cards with multimodal tags. After the user adds new multimodal information, the back end parses the new information, stores it, obtains its access address, determines its multimodal tags, and adds these tags to the corresponding existing knowledge cards. The back end then adds the access address and multimodal tags to the multimodal information's mapping table.

[0347] During the front-end knowledge card display process, the back-end determines the access address of the multimodal information corresponding to the multimodal tag in each knowledge card based on the multimodal information mapping table, and finds the multimodal information to add to the layout position of the multimodal tag in each knowledge card in the access address. Users can then view each knowledge card on the front-end display interface.

[0348] After obtaining each knowledge card, a multimodal knowledge base can be generated. The backend can vectorize each knowledge card to obtain the knowledge vector and index of each knowledge card.

[0349] In the process of acquiring and outputting multimodal information, the user inputs interactive information, and the backend combines the knowledge vectors of each knowledge card with the semantic features of the interactive information to obtain the target knowledge card. Then, by combining the mapping table of multimodal information, the target access address of the target multimodal information in the target knowledge card is obtained. Finally, the text content of the target knowledge card and the target multimodal information are presented, and the user can view the specific content of the target knowledge card through the display interface.

[0350] Based on the above embodiments, such as Figure 5 As shown in the figure, this embodiment of the invention also provides a digital human multimodal interaction method, including:

[0351] S31, Based on the multimodal information retrieval enhancement generation method provided in the above embodiments, determine the response content;

[0352] S32, Based on the response content, generate digital human driving data; the digital human driving data is used to drive the digital human to respond to interactive information.

[0353] Specifically, the digital human multimodal interaction method provided in this embodiment of the invention is executed by a digital human multimodal interaction device, which can be configured in a computer. The computer can be a local computer or a cloud computer. The local computer can be a computer, tablet, etc., and no specific limitation is made here.

[0354] First, step S31 is executed, utilizing the multimodal information retrieval enhancement generation method provided in the above embodiments to determine the response content corresponding to the interaction information of the fourth user. This interaction information can be relevant information used to drive the digital human, such as voice or text. The obtained response content contains multimodal information and text content.

[0355] Then, step S32 is executed to generate digital human driving data using the response content. This digital human driving data may include voice-driven data and motion-driven data, with the motion-driven data including facial expressions and body movement data. Specific methods for generating digital human driving data based on the response content include speech synthesis technology, speech-animation synthesis technology, video generation technology, etc., which will not be elaborated here.

[0356] By using data driven by digital humans, we can instruct digital humans to perform corresponding operations according to the instructions of interactive information. Here, the role of data driven by digital humans is to drive digital humans to use multimodal information to respond to interactive information.

[0357] The digital human multimodal interaction method provided in this embodiment of the invention first determines the response content corresponding to the interaction information; based on the response content, digital human driving data is generated to drive the digital human to respond to the interaction information. This method can drive the digital human to display response content containing multimodal information to the user, enabling the digital human to respond to any content input by the user based on the multimodal information pre-stored in the multimodal knowledge base. This not only makes the interaction process more personalized and diverse, but also ensures the authenticity and reliability of the response content.

[0358] like Figure 6 As shown, based on the above embodiments, this embodiment of the invention provides a modal knowledge base generation device, including:

[0359] File acquisition module 61 is used to acquire target files;

[0360] The file parsing module 62 is used to parse the target file and obtain the parsing result; the parsing result includes the text content and multimodal information in the target file;

[0361] The fusion module 63 is used to determine the multimodal labels of the multimodal information and fuse the text content with the multimodal labels to obtain the fusion result;

[0362] The knowledge splitting module 64 is used to split the fusion result into knowledge cards, form at least one knowledge card, and generate a multimodal knowledge base based on at least one knowledge card.

[0363] Based on the above embodiments, a mapping table construction module is also included, used for:

[0364] Determine the access address for multimodal information;

[0365] A mapping table for multimodal information is constructed based on multimodal tags and access addresses.

[0366] Based on the above embodiments, an access module is also included, used for:

[0367] If a knowledge base access request is received, the multimodal label in the first knowledge card corresponding to the knowledge base access request is replaced with the corresponding access address based on the mapping table to obtain the access result, and the access result is displayed on the access interface.

[0368] Based on the above embodiments, an editing module is also included, used for:

[0369] Receive edit requests for the second knowledge card in the multimodal knowledge base;

[0370] Edit the second knowledge card based on the edit request.

[0371] Based on the above embodiments, the editing module is specifically used for:

[0372] If the editing request is to add target multimodal information, then determine the target multimodal tag and target access address of the target multimodal information;

[0373] Store the target multimodal label and target access address into the mapping table, and add the target multimodal label to the second knowledge card.

[0374] Based on the above embodiments, the mapping table construction module is specifically used for:

[0375] Extracting summaries of multimodal information;

[0376] A mapping table is constructed based on multimodal tags, access addresses, and summaries.

[0377] Based on the above embodiments, the text content includes content fragments, which are determined based on the file type of the target file.

[0378] Based on the above embodiments, a content fragment determination module is also included, used for:

[0379] In response to a file of type text, the text content of the text file is treated as a content fragment;

[0380] In response to a file type of demo file, the text content of the demo file is divided into one or more content fragments based on the page content of the demo file;

[0381] In response to a file type of lightweight text file, the text content of the lightweight text file is divided into one or more content segments according to the minimum heading of the lightweight text file;

[0382] In response to a file type of data file, the text content of the data file is divided into one or more content fragments based on the layout format of the data file.

[0383] Based on the above embodiments, the data file is specifically a table file;

[0384] The content fragment determination module is specifically used for:

[0385] If the first row of the table file is the header row, then each row in the table file other than the first row is treated as a content fragment, and the column content in each content fragment includes the column headers in the header row.

[0386] If the first row of the table file is not the header row, then each row in the table file will be treated as a content segment.

[0387] Based on the above embodiments, the knowledge splitting module is specifically used for:

[0388] Determine the split delimiter and knowledge card length threshold;

[0389] Based on the splitting delimiter and the knowledge card length threshold, the fusion result is split into a first set of knowledge fragments;

[0390] Based on the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold, at least one knowledge card is obtained by applying the first knowledge fragment set.

[0391] Based on the above embodiments, the knowledge splitting module is also specifically used for:

[0392] For any knowledge fragment in the first set of knowledge fragments, if the length of any knowledge fragment does not exceed the knowledge card length threshold, then any knowledge fragment is treated as a knowledge card.

[0393] If the length of any knowledge fragment exceeds the knowledge card length threshold, then the knowledge fragment is split into multiple first knowledge segments whose length does not exceed the knowledge card length threshold, and each first knowledge segment is treated as a knowledge card.

[0394] Based on the above embodiments, the knowledge splitting module is also specifically used for:

[0395] Break any piece of knowledge into multiple statements;

[0396] Based on the length of each statement and the knowledge card length threshold, multiple statement sets are determined from multiple statements in the order of statements, and each statement set is treated as a knowledge card;

[0397] The length of each set of statements does not exceed the knowledge card length threshold.

[0398] Based on the above embodiments, the knowledge splitting module is also specifically used for:

[0399] Determine the knowledge redundancy length threshold, and based on the current knowledge card, apply the knowledge redundancy length threshold to identify redundant knowledge;

[0400] Copy redundant knowledge to the starting position of the next knowledge card after the current knowledge card.

[0401] Based on the above embodiments, the knowledge splitting module is also specifically used for:

[0402] Determine whether there is a second knowledge segment at the end of the current knowledge card that does not exceed the knowledge redundancy length threshold;

[0403] If there is no second knowledge segment at the end of the current knowledge card, then the last statement of the current knowledge card will be regarded as redundant knowledge.

[0404] If a second knowledge segment exists at the end of the current knowledge card, redundant knowledge is determined based on the length of the second knowledge segment and the length of the next knowledge card after the current knowledge card.

[0405] Based on the above embodiments, the knowledge splitting module is also specifically used for:

[0406] Extract title information and topic transition points from the fusion results;

[0407] Based on the title information and topic shift points, the fusion result is split into a second set of knowledge fragments;

[0408] Based on the title information and the second set of knowledge fragments, at least one knowledge card is determined.

[0409] Based on the above embodiments, the knowledge splitting module is also specifically used for:

[0410] Based on the title information, determine the initial set of knowledge fragments for the fusion result;

[0411] If it is determined that there is a specified knowledge fragment in the initial knowledge fragment set whose length exceeds the first preset length, and there is a topic transfer point in the specified knowledge fragment, then the specified knowledge fragment is segmented to obtain the segmentation result. The non-specified knowledge fragments in the initial knowledge fragment set and the segmentation result are merged to obtain an intermediate knowledge fragment set.

[0412] If it is determined that there are two adjacent knowledge fragments with the same title level in the intermediate knowledge fragment set, and the length of at least one of the two adjacent knowledge fragments is less than the second preset length, then the two adjacent knowledge fragments are merged to obtain the second knowledge fragment set.

[0413] Based on the above embodiments, an analysis module is also included, used for:

[0414] If a sharing request for a third knowledge card is received, determine the target account corresponding to the sharing request;

[0415] In response to the target account's request to allow sharing, a third knowledge card is generated in the target account's multimodal knowledge base.

[0416] Specifically, the functions of each module in the modal knowledge base generation device provided in this embodiment of the invention correspond one-to-one with the operation flow of each step in the above method-like embodiments, and the achieved effects are also the same. For details, please refer to the above embodiments, and this will not be repeated in this embodiment of the invention.

[0417] Based on the above embodiments, such as Figure 7 As shown, this embodiment of the invention also provides a multimodal information retrieval enhancement generation device, comprising:

[0418] Interaction module 71 is used to acquire interaction information;

[0419] The matching module 72 is used to extract the semantic features of the interaction information and, based on the semantic features, determine the target knowledge card in the multimodal knowledge base that matches the semantic features of the interaction information.

[0420] The response content generation module 73 is used to generate response content corresponding to interactive information based on the target knowledge card and by applying a large language model.

[0421] The multimodal knowledge base is generated based on the multimodal knowledge base generation method provided in the above embodiments.

[0422] Based on the above embodiments, such as Figure 8 As shown in the figure, this embodiment of the invention also provides a digital human multimodal interaction system, including:

[0423] The response content determination module 81 is used to determine the response content based on the multimodal information retrieval enhancement generation method provided in the above embodiments;

[0424] The digital human driving module 82 is used to generate digital human driving data based on the response content; the digital human driving data is used to drive the digital human to respond to interactive information.

[0425] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions stored in the memory 830 to execute the multimodal knowledge base generation method, the multimodal information retrieval enhancement generation method, or the digital human multimodal interaction method provided in the above embodiments.

[0426] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0427] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multimodal knowledge base generation method, or the multimodal information retrieval enhancement generation method, or the digital human multimodal interaction method provided in the above embodiments.

[0428] In another aspect, the present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multimodal knowledge base generation method, or the multimodal information retrieval enhancement generation method, or the digital human multimodal interaction method provided in the above embodiments. This computer-readable storage medium can be either a non-transitory computer-readable storage medium or a transient computer-readable storage medium; no specific limitation is made here.

[0429] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0430] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0431] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a multimodal knowledge base, characterized in that, include: Obtain the target file; The target file is parsed to obtain the parsing results; The parsing results include the text content and multimodal information in the target file; Determine the multimodal tags of the multimodal information, and fuse the text content with the multimodal tags to obtain a fusion result; The fusion result is split into knowledge segments to form at least one knowledge card, and a multimodal knowledge base is generated based on at least one of the knowledge cards. The step of performing knowledge decomposition on the fusion result to form at least one knowledge card includes: Using the delimiters in the fusion result, combined with semantic coherence detection technology, the boundaries of knowledge units in the fusion result are identified; each knowledge unit in the fusion result constitutes a knowledge card; each knowledge card includes text content and multimodal tags; Determine a knowledge redundancy length threshold, and based on the current knowledge card, apply the knowledge redundancy length threshold to determine redundant knowledge; The redundant knowledge is copied to the starting position of the next knowledge card after the current knowledge card.

2. The multimodal knowledge base generation method according to claim 1, characterized in that, The process of parsing the target file to obtain the parsing result includes: Determine the access address of the multimodal information; Based on the multimodal tags and the access addresses, a mapping table for the multimodal information is constructed.

3. The multimodal knowledge base generation method according to claim 2, characterized in that, The step of generating a multimodal knowledge base based on at least one of the knowledge cards then includes: If a knowledge base access request is received, the multimodal label in the first knowledge card corresponding to the knowledge base access request is replaced with the corresponding access address based on the mapping table to obtain the access result, and the access result is displayed on the access interface.

4. The multimodal knowledge base generation method according to claim 2, characterized in that, The process of generating a multimodal knowledge base based on at least one of the knowledge cards further includes: Receive an edit request for the second knowledge card in the multimodal knowledge base; Based on the edit request, the second knowledge card is edited.

5. The multimodal knowledge base generation method according to claim 4, characterized in that, The step of editing the second knowledge card based on the editing request includes: If the editing request is to add target multimodal information, then determine the target multimodal tag and target access address of the target multimodal information; The target multimodal tag and the target access address are stored in the mapping table, and the target multimodal tag is added to the second knowledge card.

6. The multimodal knowledge base generation method according to claim 2, characterized in that, The step of constructing a mapping table for the multimodal information based on the multimodal tag and the access address includes: Extract a summary of the multimodal information; The mapping table is constructed based on the multimodal tag, the access address, and the digest.

7. The multimodal knowledge base generation method according to claim 1, characterized in that, The text content includes content fragments, which are determined based on the file type of the target file.

8. The multimodal knowledge base generation method according to claim 7, characterized in that, The steps for determining the content fragment include: In response to the file type being a text file, the text content of the text file is treated as a content fragment; In response to the file type being a demo file, the text content of the demo file is segmented into one or more content fragments based on the page content of the demo file. In response to the file type being a lightweight text file, the text content of the lightweight text file is segmented into one or more content fragments according to the minimum heading of the lightweight text file; In response to the file type being a data file, the text content of the data file is segmented into one or more content fragments based on the layout format of the data file.

9. The multimodal knowledge base generation method according to claim 8, characterized in that, The data file is specifically a table file; The method of dividing the text content of the data class file into one or more content segments based on the layout format of the data class file includes: If the first row of the table file is the header row, then each row in the table file other than the first row is treated as a content segment, and the column content in each content segment is determined to include the column header in the header row; If the first row of the table file is not a header row, then each row in the table file will be treated as a content segment.

10. The multimodal knowledge base generation method according to any one of claims 1-9, characterized in that, The step of performing knowledge decomposition on the fusion result to form at least one knowledge card includes: Determine the split delimiter and knowledge card length threshold; Based on the splitting delimiter and the knowledge card length threshold, the fusion result is split into a first set of knowledge fragments; Based on the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold, the first knowledge fragment set is applied to obtain at least one knowledge card.

11. The multimodal knowledge base generation method according to claim 10, characterized in that, The step of applying the first knowledge fragment set to obtain at least one knowledge card based on the length of each knowledge fragment in the first knowledge fragment set and the knowledge card length threshold includes: For any knowledge fragment in the first set of knowledge fragments, if the length of any knowledge fragment does not exceed the knowledge card length threshold, then any knowledge fragment is treated as a knowledge card. If the length of any knowledge fragment exceeds the knowledge card length threshold, then the knowledge fragment is split into multiple first knowledge segments with a length not exceeding the knowledge card length threshold, and each first knowledge segment is used as a knowledge card.

12. The multimodal knowledge base generation method according to claim 11, characterized in that, The step of splitting any knowledge fragment into multiple first knowledge segments with a length not exceeding the knowledge card length threshold includes: Break any of the knowledge fragments into multiple statements; Based on the length of each statement and the knowledge card length threshold, multiple statement sets are determined from the multiple statements in statement order, and each statement set is used as a knowledge card; The length of each set of statements does not exceed the knowledge card length threshold.

13. The multimodal knowledge base generation method according to any one of claims 1-9, characterized in that, The step of determining redundant knowledge based on the current knowledge card and applying the knowledge redundancy length threshold includes: Determine whether there is a second knowledge segment at the end of the current knowledge card that does not exceed the knowledge redundancy length threshold; If the second knowledge fragment does not exist at the end of the current knowledge card, then the last statement of the current knowledge card is taken as the redundant knowledge. If the second knowledge fragment exists at the end of the current knowledge card, then the redundant knowledge is determined based on the length of the second knowledge fragment and the length of the next knowledge card after the current knowledge card.

14. The multimodal knowledge base generation method according to any one of claims 1-9, characterized in that, The step of performing knowledge decomposition on the fusion result to form at least one knowledge card also includes: Extract the title information and topic transition points from the fusion result; Based on the title information and the topic shift point, the fusion result is divided into a second set of knowledge fragments; Based on the title information and the second set of knowledge fragments, at least one of the knowledge cards is determined.

15. The multimodal knowledge base generation method according to claim 14, characterized in that, The step of splitting the fusion result into a second set of knowledge fragments based on the title information and the topic transition point includes: Based on the title information, determine the initial set of knowledge fragments for the fusion result; If it is determined that there is a specified knowledge fragment in the initial knowledge fragment set whose length exceeds the first preset length, and there is a topic transfer point in the specified knowledge fragment, then the specified knowledge fragment is segmented to obtain the segmentation result, and the non-specified knowledge fragments in the initial knowledge fragment set and the segmentation result are merged to obtain an intermediate knowledge fragment set. If it is determined that there are two adjacent knowledge fragments with the same title level in the intermediate knowledge fragment set, and the length of at least one of the two adjacent knowledge fragments is less than the second preset length, then the two adjacent knowledge fragments are merged to obtain the second knowledge fragment set.

16. The multimodal knowledge base generation method according to any one of claims 1-9, characterized in that, The process of generating a multimodal knowledge base based on at least one of the knowledge cards further includes: If a sharing request for a third knowledge card is received, determine the target account corresponding to the sharing request; In response to the target account granting the sharing request, the third knowledge card is generated in the target account's multimodal knowledge base.

17. A multimodal information retrieval enhancement generation method, characterized in that, include: Obtain interactive information; Extract the semantic features of the interaction information, and based on the semantic features, determine the target knowledge card in the multimodal knowledge base that matches the semantic features of the interaction information; Based on the target knowledge card, a large language model is applied to generate the response content corresponding to the interactive information; The multimodal knowledge base is generated based on the multimodal knowledge base generation method as described in any one of claims 1-16.

18. The multimodal information retrieval enhancement generation method according to claim 17, characterized in that, The step of determining the target knowledge card in the multimodal knowledge base that matches the semantic features of the interaction information based on the semantic features includes: Determine the knowledge vector of each knowledge card in the multimodal knowledge base; The target knowledge card is determined based on the similarity between the semantic features and the knowledge vector.

19. The multimodal information retrieval enhancement generation method according to claim 17, characterized in that, The response content contains a target multimodal tag, and the target multimodal tag corresponds to a target access address; The step of generating response content corresponding to the interactive information based on the target knowledge card and applying a large language model includes: Based on the target access address, the target multimodal information in the response content is determined, and the response content is displayed on the interactive interface.

20. A digital human multimodal interaction method, characterized in that, include: The response content is determined based on the multimodal information retrieval enhancement generation method as described in any one of claims 17-19; Based on the response content, digital human driving data is generated; The digital human driving data is used to drive the digital human to respond to the interactive information.

21. A modal knowledge base generation device, characterized in that, include: The file acquisition module is used to acquire the target file; The file parsing module is used to parse the target file and obtain the parsing result; The parsing results include the text content and multimodal information in the target file; The fusion module is used to determine the multimodal tags of the multimodal information and fuse the text content with the multimodal tags to obtain a fusion result; The knowledge decomposition module is used to decompose the fusion result into knowledge cards, and generate a multimodal knowledge base based on at least one of the knowledge cards. The knowledge decomposition module is specifically used for: Using the delimiters in the fusion result, combined with semantic coherence detection technology, the boundaries of knowledge units in the fusion result are identified; each knowledge unit in the fusion result constitutes a knowledge card; each knowledge card includes text content and multimodal tags; Determine a knowledge redundancy length threshold, and based on the current knowledge card, apply the knowledge redundancy length threshold to determine redundant knowledge; The redundant knowledge is copied to the starting position of the next knowledge card after the current knowledge card.

22. A multimodal information retrieval enhancement generation device, characterized in that, include: The interaction module is used to obtain interaction information; The matching module is used to extract the semantic features of the interaction information and, based on the semantic features, determine the target knowledge card in the multimodal knowledge base that matches the semantic features of the interaction information. The response content generation module is used to generate response content corresponding to the interactive information based on the target knowledge card and by applying a large language model. The multimodal knowledge base is generated based on the multimodal knowledge base generation method as described in any one of claims 1-16.

23. A digital human multimodal interaction system, characterized in that, include: The response content determination module is used to determine the response content based on the multimodal information retrieval enhancement generation method as described in any one of claims 17-19; A digital human driving module is used to generate digital human driving data based on the response content; The digital human driving data is used to drive the digital human to respond to the interactive information.

24. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multimodal knowledge base generation method as described in any one of claims 1-16, or the multimodal information retrieval enhancement generation method as described in any one of claims 17-19, or the digital human multimodal interaction method as described in claim 20.

25. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal knowledge base generation method as described in any one of claims 1-16, or the multimodal information retrieval enhancement generation method as described in any one of claims 17-19, or the digital human multimodal interaction method as described in claim 20.

26. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal knowledge base generation method as described in any one of claims 1-16, or the multimodal information retrieval enhancement generation method as described in any one of claims 17-19, or the digital human multimodal interaction method as described in claim 20.

Images