Multi-role digital human construction method based on multi-modal graph retrieval enhancement generation

Through multi-modal graph retrieval enhancement generation technology, the external knowledge base is built and the large language model is integrated, which solves the problem of time-consuming and labor-intensive construction of digital humans and the inability to acquire deep knowledge, and realizes high-quality Q&A interaction and rapid switching of multiple roles. Improves the accuracy of user experience and Q&A content.

CN119003746AInactive Publication Date: 2024-11-22ZHEJIANG LOTUS PURPLE STAR INTELLIGENT COMPUTING TECHNOLOGY CO LTD

Patent Information

Application Number
CN202411488644.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-24
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing digital human construction method requires a lot of time to prepare local professional knowledge bases and fine-tuning training of models. It is time-consuming and labor-intensive, and has high application costs. It cannot allow digital humans to obtain deep knowledge outside the field, which is prone to hallucinations. The outputted Q&A content is of poor quality.

Method used

Using a method based on multi-modal graph retrieval enhancement generation to build an external knowledge base, integrate a large language model, and assist digital people in question-and-answer questions and answers through multi-modal processing to realize the interaction between digital people in specific language styles and domain knowledge, reduce training costs, and increase training costs Q&A quality and support for quick switching of multiple characters.

Benefits of technology

Improve the quality and richness of Q&A content, reduce the training cost of large language models, realize the unity of digital human image and domain knowledge, enhance user experience, and provide personalized feedback in complex scenarios to meet different scenarios and users. need.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003746A_ABST
    Figure CN119003746A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of digital human interaction, in particular to a multi-role digital human construction method based on multi-modal graph retrieval enhancement generation, which comprises the following steps of: S1, constructing an external knowledge base by adopting a graph retrieval enhancement generation technology, and constructing a large language model for performing multi-modal processing; s2, creating a digital human interface for inputting external information; and S3, establishing a digital human framework, and integrating a large language model and an external knowledge base to perform digital human question and answer work. According to the method, an external knowledge base is provided for a large language model through a graph retrieval enhancement generation technology, digital human question and answer work is assisted in a multi-modal processing mode, the function that digital human interacts with specific field knowledge in a specific language style can be achieved, and therefore the output question and answer content quality is improved, and the user experience is improved. The training cost of the large language model can be reduced, the question and answer quality can be improved, and multi-role fast switching of the digital person can be carried out.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of digital human interaction technology, and in particular to a method for constructing a multi-role digital human based on multimodal graph retrieval enhanced generation. Background Art

[0002] With the rapid development of generative AI technologies such as ChatGPT, digital humans driven by these large language models have been widely used in multiple industries, including personal assistance, intelligent customer service, and online education. These anthropomorphic digital humans can quickly interact with users, significantly enhancing the user experience of human-computer interaction. However, digital humans of different roles need to have knowledge in different fields and use different language styles when interacting. General large-scale language models usually cannot cover all knowledge in a specific field. Therefore, current multi-role digital human generation solutions usually rely on manual presets, such as a knowledge base text matching question and answer method, device and equipment based on a dialogue model disclosed in a Chinese patent (publication number: CN118427326A). In this patented technology, historical operation and maintenance event work orders are classified to obtain classification results, and consulting work orders are screened according to the classification results. Business knowledge points that meet preset business rules are extracted from consulting work orders to obtain a knowledge point set. A local professional knowledge base is constructed based on the knowledge point set to perform question and answer work. However, both the construction of a local professional knowledge base and fine-tuning training will take a lot of time, be time-consuming and labor-intensive, and have a high application cost. In addition, the digital human cannot obtain deep knowledge outside the field, which is prone to hallucination problems, and the output question and answer content quality is poor. Summary of the invention

[0003] Technical problem to be solved by the present invention: The existing method of constructing digital humans requires a lot of time to prepare a local professional knowledge base and fine-tune the model, which is time-consuming and labor-intensive, has a high application cost, and cannot allow digital humans to acquire deep knowledge outside the field, is prone to hallucination problems, and the quality of the output question and answer content is poor.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solution: a method for constructing a multi-role digital human based on multimodal graph retrieval enhanced generation, comprising the following steps: S1: Use graph retrieval enhancement generation technology to build an external knowledge base and a large language model for multimodal processing; S2: Create a digital human interface for external information input; S3: Establish a digital human architecture, integrate large language models and external knowledge bases to perform digital human question-answering work.

[0005] When the present invention is working, it provides an external knowledge base for the large language model through graph retrieval enhancement generation technology, and adopts multimodal processing to assist the digital human question and answer work, which can realize the function of digital humans interacting in a specific language style using specific domain knowledge, thereby improving the quality of the output question and answer content, reducing the training cost of the large language model, improving the question and answer quality, and enabling rapid switching of multiple roles of digital humans.

[0006] Preferably, in step S1, when the external knowledge base is constructed by using graph retrieval enhancement generation technology, the following steps are adopted: A1: Collect external files containing multimodal information; A2: Preprocess the multimodal information in the file, set the multimodal information or preset information points in the file as graph nodes, generate edges through semantic matching to connect the corresponding graph nodes, so as to form a multimodal knowledge graph, thereby obtaining a graph structure; A3: Establish an index for the constructed knowledge graph, store the graph structure and the corresponding index as an external knowledge base, and use graph retrieval enhancement generation technology for continuous updating.

[0007] When the present invention is working, it obtains relevant information from a wide range of documents through graph retrieval enhancement generation technology, and can organize it into a graph structure, which is convenient for efficient knowledge integration and dissemination, and is convenient for digital humans to fully consider multimodal information in question and answer, which can improve the richness and accuracy of question and answer, and facilitate digital humans to switch roles, realize the unification of digital humans' image, domain knowledge and language style, and can effectively improve the user experience.

[0008] Preferably, in step A1, the source of the file includes at least one of a network open document library, a professional domain knowledge library and an internal enterprise knowledge library, and the type of multimodal information includes at least one of an image, text, video and audio.

[0009] Preferably, in step A2, when preprocessing the multimodal information in the file, the following steps are included: B1: Call the preset image processing model, determine the corresponding perceptual conversion method according to the type of multimodal information, and convert the multimodal information that meets the requirements into a high-dimensional vector representation that meets the constraints of the same vector space; B2: The converted high-dimensional vector representation is input into the same vector space for semantic matching in accordance with the vector calculation rules, and the multimodal information corresponding to the high-dimensional vector representations with similar distances is marked as semantically similar or related information.

[0010] Preferably, the image processing model is configured to be CLIP for converting different types of multimodal information into a high-dimensional vector representation that can be represented in the same vector space.

[0011] When the present invention is working, by preprocessing the multimodal image in the file, it is convenient to combine the multimodal data processing capability and the graph structure optimization, so that the external knowledge base is rich and complete, so that the digital human can also understand the user's needs well in complex scenes and provide more personalized feedback, so as to meet different scenes and user needs. At the same time, CLIP is used to convert and encode multimodal information, which can shorten the distance between semantically similar or related information in the vector space, thereby improving the multimodal information expression capability of the graph structure with high accuracy.

[0012] Preferably, in step A4, the index of the graph structure includes at least one of a label, a summary, a keyword of a corresponding graph node in the graph structure, and a high-dimensional vector representation of the graph node.

[0013] Preferably, in step S1, when constructing a large language model for multimodal processing, the following steps are adopted: C1: Select the corresponding pre-trained language model based on the preset project requirements and available resources; C2: Based on the preset project requirements and available resources, obtain a relevant data set to cover the language and context of the project requirements; C3: Define tasks and goals, modify the pre-trained language model and perform fine-tuning training. The fine-tuned pre-trained language model is output as a large language model after passing the performance evaluation.

[0014] Preferably, in step C3, the pre-trained language model is modified and then fine-tuned, and when the fine-tuned pre-trained language model is output as a large language model after passing performance verification, the following steps are adopted: the pre-trained language model is fine-tuned using a data set, the parameters of the pre-trained language model are updated through a back propagation algorithm to maximize the performance on the target task data set, the model performance is evaluated at the end of each training cycle, and the hyperparameters are adjusted for optimization, and the model is output as a large language model after passing the performance evaluation.

[0015] Preferably, in step S2, the input external information includes at least one of text input, voice input, image input and video input.

[0016] Preferably, in step S3, when establishing a digital human architecture and integrating a large language model and an external knowledge base to perform digital human question and answer work, the following steps are adopted: integrating the large language model and the external knowledge base into the digital human architecture, and performing digital human question and answer work through multimodal processing and graph retrieval enhanced generation technology.

[0017] The beneficial technical effects of the present invention include: 1. The present invention provides an external knowledge base for the large language model through graph retrieval enhancement generation technology, and adopts multimodal processing to assist the digital human question and answer work, which can realize the function of digital humans interacting in a specific language style using specific domain knowledge, thereby improving the quality of the output question and answer content, reducing the training cost of the large language model, improving the question and answer quality, and enabling rapid switching of multiple roles of digital humans.

[0018] 2. The present invention obtains relevant information from a wide range of documents through graph retrieval enhancement generation technology, and can organize it into a graph structure, which is convenient for efficient knowledge integration and dissemination, and helps digital humans to fully consider multimodal information in question and answer, which can improve the richness and accuracy of question and answer, and facilitate digital humans to switch roles, so as to achieve the unification of digital humans' image, domain knowledge and language style, and effectively improve the user experience.

[0019] 3. The present invention pre-processes the multimodal images in the file, conveniently combining the multimodal data processing capabilities and graph structure optimization, so that the external knowledge base is rich and complete, and the digital human can also understand the user's needs well and provide more personalized feedback in complex scenes, thereby meeting different scenes and user needs. At the same time, CLIP is used to convert and encode multimodal information, which can shorten the distance between semantically similar or related information in the vector space, thereby improving the multimodal information expression capability of the graph structure with high accuracy.

[0020] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The present invention will be further described below in conjunction with the accompanying drawings: Attached Figure 1 Workflow diagram of the method for building multi-role digital humans based on multimodal graph retrieval-enhanced generation. DETAILED DESCRIPTION

[0022] The technical solutions of the embodiments of the present invention are explained and described below in conjunction with the drawings of the embodiments of the present invention, but the following embodiments are only preferred embodiments of the present invention, not all. Based on the embodiments in the implementation mode, other embodiments obtained by those skilled in the art without creative work are all within the protection scope of the present invention.

[0023] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the referred device or element must have a specific direction, be constructed and operate in a specific direction. Therefore, it should not be understood as a limitation of the present invention. Embodiment 1:

[0024] Please see attached Figure 1 This embodiment discloses a method for constructing a multi-role digital human based on multi-modal graph retrieval and enhanced generation, and the method for constructing a multi-role digital human based on multi-modal graph retrieval and enhanced generation includes the following steps: S1: Use graph retrieval enhancement generation technology to build an external knowledge base and a large language model for multimodal processing; S2: Create a digital human interface for external information input; S3: Establish a digital human architecture, integrate large language models and external knowledge bases to perform digital human question-answering work.

[0025] When this embodiment is working, an external knowledge base is provided for the large language model through graph retrieval enhancement generation technology, and multimodal processing is used to assist the digital human question and answer work, which can realize the function of digital humans interacting in a specific language style and using specific domain knowledge, thereby improving the quality of the output question and answer content, reducing the training cost of the large language model, improving the question and answer quality, and enabling rapid switching of multiple roles of digital humans.

[0026] Preferably, in order to further improve the performance of the constructed digital human, the large language model may be fine-tuned and trained. In step S1, when constructing the large language model for multimodal processing, the following steps are adopted: C1: Select the corresponding pre-trained language model according to the preset project requirements and available resources. In the specific implementation, you can choose the appropriate model according to the actual needs, such as BERT or Llama. C2: Based on the preset project requirements and available resources, obtain a relevant dataset to cover the language and context required by the project. In the specific implementation, the dataset should contain enough samples to meet the language and context required by the task. For example, a large amount of multi-round dialogue data in Chinese and English can be input; C3: Define tasks and goals, such as text classification, machine translation, and question-answering systems, and modify the pre-trained language model for fine-tuning. For example, you can add an appropriate classification layer to the BERT model to suit classification tasks. The fine-tuned pre-trained language model is output as a large language model after passing performance evaluation.

[0027] As a further improvement of this embodiment, in step C3, the pre-trained language model is modified and then fine-tuned. When the fine-tuned pre-trained language model is output as a large language model after passing performance verification, the following steps are adopted: the pre-trained language model is fine-tuned using a data set, the parameters of the pre-trained language model are updated through a back propagation algorithm to maximize the performance on the target task data set, the model performance is evaluated at the end of each training cycle, and the hyperparameters are adjusted for optimization, and the pre-trained language model is output as a large language model after passing the performance evaluation.

[0028] Preferably, in step S2, the input external information includes at least one of text input, voice input, image input and video input. In specific implementation, voice-to-text, text-to-speech or lip synchronization input methods can also be performed to facilitate user use.

[0029] In the specific implementation, in step S3, when establishing a digital human architecture, integrating a large language model and an external knowledge base to perform digital human question and answer work, the following steps are adopted: integrating the large language model and the external knowledge base into the digital human architecture, and performing digital human question and answer work through multimodal processing and graph retrieval enhanced generation technology. Embodiment 2:

[0030] Please see attached Figure 1 This embodiment provides a method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation. The similarities with other embodiments will not be repeated here, and the differences will be described in detail below.

[0031] In this embodiment, in step S1, when the graph retrieval enhancement generation technology is used to construct the external knowledge base, the following steps are adopted: A1: Collect external files containing multimodal information; A2: Preprocess the multimodal information in the file, set the multimodal information or preset information points in the file as graph nodes, generate edges through semantic matching to connect the corresponding graph nodes, so as to form a multimodal knowledge graph, thereby obtaining a graph structure; A3: Establish an index for the constructed knowledge graph, store the graph structure and the corresponding index as an external knowledge base, and use graph retrieval enhancement generation technology for continuous updating.

[0032] When this embodiment is working, it obtains relevant information from a wide range of documents through graph retrieval enhancement generation technology and can organize it into a graph structure, which is convenient for efficient knowledge integration and dissemination, and is convenient for digital humans to fully consider multimodal information in question and answer, which can improve the richness and accuracy of question and answer, and facilitate digital humans to switch roles, realize the unification of digital humans' image, domain knowledge and language style, and can effectively improve the user experience.

[0033] In the specific implementation, in step A1, the source of the file includes at least one of an open document library on the Internet, a professional knowledge base and an internal knowledge base of the enterprise, and the type of multimodal information includes at least one of an image, text, video and audio. Preferably, the knowledge field of the file can include relevant fields such as medicine, technology and law.

[0034] Preferably, in step A2, when preprocessing the multimodal information in the file, the following steps are included: B1: Call the preset image processing model, determine the corresponding perceptual conversion method according to the type of multimodal information, and convert the multimodal information that meets the requirements into a high-dimensional vector representation that meets the constraints of the same vector space; B2: The converted high-dimensional vector representation is input into the same vector space for semantic matching in accordance with the vector calculation rules, and the multimodal information corresponding to the high-dimensional vector representations with similar distances is marked as semantically similar or related information.

[0035] As a further improvement of this embodiment, the image processing model is configured as CLIP for converting different types of multimodal information into high-dimensional vector representations that can be represented in the same vector space. In specific implementation, CLIP can jointly train the representations of text and images in multimodal information by contrastive learning, so that they can be compared and distance calculated in the same vector space, thereby facilitating the marking of semantically similar or related information. Preferably, CLIP can also be fine-tuned, and back propagation and gradient descent methods can be used to further optimize CLIP to make its performance more excellent so as to be suitable for processing different types of multimodal information in different fields.

[0036] When this embodiment is working, by preprocessing the multimodal images in the file, it is convenient to combine the multimodal data processing capabilities and graph structure optimization, so that the external knowledge base is rich and complete, so that the digital human can also understand the user's needs well and provide more personalized feedback in complex scenarios, thereby meeting different scenarios and user needs. At the same time, CLIP is used to convert and encode multimodal information, which can shorten the distance between semantically similar or related information in the vector space, thereby improving the multimodal information expression capability of the graph structure with high accuracy.

[0037] Preferably, in step A4, the index of the graph structure includes at least one of a label, a summary, a keyword of a corresponding graph node in the graph structure, and a high-dimensional vector representation of the graph node. Embodiment three:

[0038] Please see attached Figure 1This embodiment provides a method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation. The similarities with other embodiments will not be repeated here, and the differences will be described in detail below.

[0039] In this embodiment, in step S3, when performing digital human question-answering work through multimodal processing and graph retrieval enhancement generation technology, the following steps are adopted: D1: Get external information input by the user; D2: Extract user questions from external information through a large language model; D3: After preprocessing the user's question, vector retrieval is performed in the external knowledge base using graph retrieval enhancement generation technology to obtain relevant information containing domain knowledge and image style; D4: The large model language performs knowledge fusion based on the relevant information obtained. After model reasoning, a digital human character with a suitable image style is selected for interactive question and answer based on the relevant information obtained. This can improve the quality of question and answer and facilitate the switching of digital human characters. After knowledge fusion, the answer content is obtained through large language model reasoning, which can avoid certain hallucination problems. Preferably, multiple output methods such as voice output or video output can be used, which has a good interactive effect and improves the user experience.

[0040] In specific implementation, the graph structure can also be semantically recognized through a large language model, and the corresponding classification rules can be preset to classify the graph structure in the external knowledge base into clusters with multiple different question and answer styles. In step D4, when a digital human character with a suitable image style is selected for interactive question and answer based on the relevant information obtained, the call weight of the cluster that matches the digital human character with the current image style can be increased, which can further improve the generation efficiency of interactive question and answer while improving the user experience and reducing the system load.

[0041] The beneficial technical effects of this embodiment include: the present invention provides an external knowledge base for the large language model through graph retrieval enhancement generation technology, and adopts multimodal processing to assist the digital human question and answer work, which can realize the function of digital humans interacting in a specific language style and using specific domain knowledge, thereby improving the quality of the output question and answer content, reducing the training cost of the large language model, improving the question and answer quality, and enabling rapid switching of multiple roles of digital humans.

[0042] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Those skilled in the art should understand that the present invention includes but is not limited to the contents described in the drawings and the above specific embodiments. Any modification that does not deviate from the functional and structural principles of the present invention will be included in the scope of the claims.

Claims

1. A multi-role digital human construction method based on multimodal graph retrieval and enhanced generation, characterized in that: The following steps are involved: S1: Use graph retrieval enhancement generation technology to build an external knowledge base and a large language model for multimodal processing; S2: Create a digital human interface for external information input; S3: Establish a digital human architecture, integrate large language models and external knowledge bases to perform digital human question-answering work.

2. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 1, characterized in that: In step S1, when the graph retrieval enhancement generation technology is used to construct the external knowledge base, the following steps are adopted: A1: Collect external files containing multimodal information; A2: Preprocess the multimodal information in the file, set the multimodal information or preset information points in the file as graph nodes, generate edges through semantic matching to connect the corresponding graph nodes, so as to form a multimodal knowledge graph, thereby obtaining a graph structure; A3: Establish an index for the constructed knowledge graph, store the graph structure and the corresponding index as an external knowledge base, and use graph retrieval enhancement generation technology for continuous updating.

3. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 2 is characterized in that: In step A1, the source of the file includes at least one of a network open document library, a professional domain knowledge library and an internal enterprise knowledge library, and the type of multimodal information includes at least one of an image, text, video and audio.

4. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 2, characterized in that: In the step A2, when preprocessing the multimodal information in the file, the following steps are included: B1: Call the preset image processing model, determine the corresponding perceptual conversion method according to the type of multimodal information, and convert the multimodal information that meets the requirements into a high-dimensional vector representation that meets the constraints of the same vector space; B2: The converted high-dimensional vector representation is input into the same vector space for semantic matching in accordance with the vector calculation rules, and the multimodal information corresponding to the high-dimensional vector representations with similar distances is marked as semantically similar or related information.

5. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 4 is characterized in that: The image processing model is configured as CLIP for converting different kinds of multimodal information into a high-dimensional vector representation that can be represented in the same vector space.

6. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 2, characterized in that: In step A4, the index of the graph structure includes at least one of a label, a summary, a keyword of a corresponding graph node in the graph structure, and a high-dimensional vector representation of the graph node.

7. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 1, characterized in that: In step S1, when constructing a large language model for multimodal processing, the following steps are adopted: C1: Select the corresponding pre-trained language model based on the preset project requirements and available resources; C2: Based on the preset project requirements and available resources, obtain a relevant data set to cover the language and context of the project requirements; C3: Define tasks and goals, modify the pre-trained language model and perform fine-tuning training. The fine-tuned pre-trained language model is output as a large language model after passing the performance evaluation.

8. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 7, characterized in that: In step C3, the pre-trained language model is modified and then fine-tuned. When the fine-tuned pre-trained language model is output as a large language model after passing performance verification, the following steps are adopted: the pre-trained language model is fine-tuned using a data set, the parameters of the pre-trained language model are updated through a back propagation algorithm to maximize the performance on the target task data set, the model performance is evaluated at the end of each training cycle, and the hyperparameters are adjusted for optimization, and the model is output as a large language model after passing the performance evaluation.

9. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 1, characterized in that: In step S2, the input external information includes at least one of text input, voice input, image input and video input.

10. The method for constructing a multi-role digital human based on multimodal graph retrieval and enhanced generation according to claim 1, characterized in that: In step S3, when establishing a digital human architecture and integrating a large language model and an external knowledge base to perform digital human question and answer work, the following steps are adopted: integrating the large language model and the external knowledge base into the digital human architecture, and performing digital human question and answer work through multimodal processing and graph retrieval enhanced generation technology.

Citation Information

Patent Citations

  • Retrieval method and system based on GPT large model generation and knowledge fusion

    CN117609589A

  • Question and answer system construction method fusing knowledge graph, knowledge base and large language model

    CN117688189A

  • Large language model knowledge question-answering method and system fused with multi-modal knowledge graph

    CN118627628A

Cited By

  • Network attack investigation method and system based on traceability graph retrieval enhancement

    CN119835023A

  • Intelligent reasoning method and system based on global search in combination with external information

    CN120068946A

  • Intelligent agent role switching method and system based on multi-modal perception and related components

    CN120197139A

  • AI digital human-based customer service platform, service method, equipment and medium

    CN120508623A