Multi-modal text travel knowledge base response method and system based on large language model

Abstract

The invention discloses a multi-modal text travel knowledge base response method and system based on a large language model. The method comprises the following steps: acquiring a text and an image input by a user; performing retrieval through an image similarity retrieval technology and a text knowledge base; and returning tourism information related to user input. According to the system architecture, a large language model, image similarity retrieval and text knowledge base construction are integrated, multi-modal input can be efficiently processed, and personalized and accurate tourism information services are achieved. The system further comprises a set of Web user interface which supports the user to interact through multi-mode query of images and texts. According to the method, the response speed and accuracy of the system can be effectively improved, and the expansibility of supporting voice interaction, multi-modal data processing and knowledge base content automatic updating is achieved. The method and the system are suitable for scenes needing to provide personalized tourism information services, and have relatively high practicability and innovativeness.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a multimodal cultural and tourism knowledge base response method and system based on a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, technologies such as large language models, image similarity retrieval, and text knowledge base have become an important force in promoting the development of fields such as information retrieval, natural language processing, and machine learning.

[0003] These technologies not only greatly enrich the ability to acquire, process and understand large-scale data sets, but also provide unprecedented intelligent solutions for all walks of life.

[0004] Especially in the field of culture and tourism, the application of these technologies has opened up new possibilities, providing users with richer and more personalized experiences, while also bringing innovative business models to the industry.

[0005] However, despite the significant progress these technologies have made in theory and application, they still face multiple challenges in actual deployment and application. How to effectively deploy large models to process and generate high-quality multimodal data, how to accurately and quickly perform image retrieval and combine it with text information to assist in content generation, and how to build and use text knowledge bases to improve the intelligence level of the system are all issues that need to be addressed.

[0006] In specific fields, such as culture and tourism, the comprehensive application of these technologies and optimization and innovation based on actual needs is also a topic worthy of in-depth discussion.

[0007] In practical applications, current technologies face the challenge of how to better integrate multimodal information and improve user experience, which is of great significance to promoting intelligent services and innovations in the cultural and tourism sector. Summary of the invention

[0008] The purpose of the present invention is to provide a multimodal cultural and tourism knowledge base response method and system based on a large language model, which is a knowledge base response method and system based on a large language model that can generate accurate answers in multimodal information queries based on a local knowledge base of text and pictures.

[0009] In order to achieve the above purpose, a multimodal cultural and tourism knowledge base answering method based on a large language model includes the following steps:

[0010] S1: Build a large language model;

[0011] S2: Build a cultural and tourism knowledge base containing text and pictures;

[0012] S3: vectorize the text knowledge in the cultural and tourism knowledge base to form a text knowledge base;

[0013] S4: vectorize the image knowledge in the cultural and tourism knowledge base to form an image knowledge base;

[0014] S5: For the image part of the user's question, query and match in the image knowledge base, obtain images related to the question, and return text description information bound to the image;

[0015] S6: Combine the text part of the user's question and the text description information bound to the image obtained in step S5, search for matches in the text knowledge base, and obtain related text data;

[0016] S7: Input the text part of the user's question and the associated text data obtained in step S6 into the large language model to generate prompt words for the large language model. The large language model generates answer content based on the prompt words.

[0017] S8: Design and deploy a web application that facilitates user access to the system.

[0018] In step S1, the large language model uses the ChatGLM model. This model was developed by Zhipu AI and has the advantages of supporting bilingualism, high precision, fast reasoning, reproducibility, and cross-platform, providing a method for building a large language model on a local server.

[0019] In step S2, a cultural tourism knowledge base including text and pictures is constructed to integrate local cultural tourism knowledge in the form of text and pictures into a database, specifically including:

[0020] First, collect and organize textual knowledge, collect relevant textual materials of famous tourist attractions, and denoise the textual materials, including removing irrelevant characters and sentence processing; second, collect and organize image knowledge, collect photos of landscape architecture, special food, etc. in relevant areas, and unify the format and resolution of the collected images. The well-organized cultural and tourism knowledge base of text and pictures provides the system with a rich knowledge reserve to deal with cultural and tourism-specific questions raised by users to the large language model.

[0021] In step S3, the text knowledge in the cultural and tourism knowledge base is vectorized. The text knowledge in the cultural and tourism knowledge base is first converted into vector form using the FastGPT text knowledge base processing system, and then stored in the PostgreSQL database, specifically including:

[0022] First, the text knowledge in the cultural and tourism knowledge base is converted into vector form using the FastGPT text knowledge base processing system, and then stored in the PostgreSQL database. The FastGPT text knowledge base processing system is developed by the open source community. It provides an interface for accessing the Moka Massive Mixed Embedding (M3E) model and providing external services to realize the text-to-vector conversion process. Among them, the M3E vector conversion model is developed by Moka (Beijing Xi Ruiyas Technology). It trains the data set with more than 22 million Chinese sentences and has good performance in the matching problem of Chinese text. The database used to store vector text is the PostgreSQL database, which is developed by the open source community. Its data storage format is relatively flexible and suitable for storing vector text.

[0023] In step S4, the image knowledge in the cultural and tourism knowledge base is vectorized. The image knowledge in the cultural and tourism knowledge base is first converted into vector form using the Towhee vectorization framework and then stored in the Milvus database, which specifically includes:

[0024] The image knowledge in the cultural and tourism knowledge base is first converted into vector form using the Towhee vectorization framework, and then stored in the Milvus database. The Towhee vectorization framework, developed by the open source community, provides an interface for calling the ResNet50 model to realize the image-to-vector conversion process. Among them, ResNet50 is a classic deep residual network model. The database used to store vector images is the Milvus vector database, developed by Zilliz. It has the characteristics of high availability, high performance, and easy expansion. It provides a solution to the problem of vector similarity retrieval, and performs particularly well in the retrieval of vector images.

[0025] Further preferably, a multimodal knowledge base answering method based on a large language model comprises:

[0026] Provide a large language model based on a pre-trained large-scale language model that can process and understand natural language input as a core component for understanding user questions and generating answers;

[0027] Build a local knowledge base containing text and images, integrating multimodal information related to local culture and tourism, including text descriptions and image data;

[0028] Vectorize the text data in the local knowledge base to generate a text vector representation that is easy to retrieve and match;

[0029] Vectorize the image data in the local knowledge base to generate embedded representations of the image to support image-based retrieval and matching;

[0030] For the image-related part in the user's question, query the image knowledge base and return the images and description information related to the user's question;

[0031] Combine the image description with the text part of the user's question and query the text knowledge base to obtain text data associated with the user's question;

[0032] Combine the obtained text data, image descriptions and user question data to generate prompt words for prompting the large language model to generate answers;

[0033] The large language model generates answers containing text and image information based on the prompt words to ensure that users receive a comprehensive and accurate response;

[0034] The web application enables users to communicate with the large model easily through a reasonable interactive interface layout.

[0035] Preferably, the vectorization processing of text and images is performed based on natural language processing models and image embedding technology.

[0036] Preferably, query matching is performed through similarity calculation, and the system performs similarity matching between user questions and text and image data in the knowledge base, and sorts them according to the matching degree, and selects the most relevant knowledge as the target data.

[0037] A system for implementing the multimodal cultural and tourism knowledge base response method based on a large language model includes:

[0038] Large language model module, used to understand user questions and generate responses;

[0039] The local knowledge base module stores text and image data related to local culture and tourism;

[0040] The text vectorization processing module is used to vectorize the text knowledge in the cultural and tourism knowledge base;

[0041] Image vectorization processing module, used to vectorize image knowledge in the cultural and tourism knowledge base;

[0042] A query module, used to retrieve text and images in the image knowledge base and the text knowledge base based on user questions;

[0043] A prompt word generation module is used to generate prompt words to guide the large language model to generate answers;

[0044] The web application module is used for users to interact with the cultural and tourism model.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] This paper proposes an innovative multimodal cultural and tourism knowledge base question-answering system, which has significant advantages over the existing large language model knowledge base question-answering technology. By building a multimodal knowledge base containing text and pictures, the system achieves deep integration of information and can more comprehensively meet the diverse needs of users in the fields of culture and tourism.

[0047] In the vectorization processing of text and images, the present invention adopts the advanced FastGPT system and Moka's M3E model for text processing, and the Towhee framework and ResNet50 model for image processing, thereby significantly improving the efficiency and accuracy of information retrieval.

[0048] In addition, by combining the text and picture information of the user's question, the system can generate more accurate answers and optimize the user's interactive experience. It is worth mentioning that the system supports deployment on local servers, which improves the speed and security of data processing, avoids dependence on cloud services, and thus better protects user privacy. Finally, the present invention is deeply customized for the specific needs of the cultural and tourism industry, providing a set of professional solutions, which significantly improves the practicality and effectiveness of information processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of the cultural tourism model method in an embodiment of the present invention.

[0050] Figure 2 It is a structural diagram of the cultural and tourism model system in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] like Figure 1 As shown, the method of the present invention is mainly divided into the following eight steps:

[0052] S1: Build a large language model;

[0053] The present invention chooses to use the ChatGLM domestic local open source large model. In order to improve efficiency and reduce costs, it is built using its acceleration technology ChatGLM.cpp. This technology reconstructs the original large model project through the C++ language, optimizes the model architecture and calculation process, and significantly reduces the demand for computing resources while maintaining model performance.

[0054] S2: Build a cultural and tourism knowledge base containing text and pictures;

[0055] The construction of the cultural and tourism knowledge base is divided into two parts: text and pictures. First, collect and organize text knowledge, collect relevant text materials from Tangqi Ancient Town, Nanxun Ancient Town, Wuzhen and other areas, denoise the text materials, including removing irrelevant characters, sentence processing and other operations, and finally build a text type knowledge base with about 70,300 words; secondly, collect and organize picture knowledge, collect photos of landscape architecture, special food and other areas in relevant areas, convert the collected pictures into JPEG format, clean up some unclear and featureless pictures, and scale them proportionally to equal-width pictures with a width of 1024 pixels, and finally build a picture type knowledge base containing 115 valid pictures. The well-organized cultural and tourism knowledge base of text and pictures provides the system with a rich knowledge reserve to deal with cultural and tourism-specific questions raised by users to the large language model.

[0056] S3: vectorize the text knowledge in the cultural and tourism knowledge base to form a text knowledge base;

[0057] The process of vectorizing text knowledge mainly uses the FastGPT text knowledge base processing system, which provides an interface for accessing the M3E vector conversion model and providing services to the outside. Among them, the FastGPT system and the M3E model can be deployed locally without a network. In order to facilitate deployment and updating, this system uses the Docker deployment method to build the FastGPT project, and deploys and sets up network communication between containers through Docker compose related commands, so that the project can run normally and access the background through the Web. For the convenience of management, this system also aligns the API interfaces of the FastGPT system, M3E model, and ChatGLM3 large model through the open source project One-API, so that the systems can work together. The storage of the text knowledge base is implemented by the PostgreSQL database, which is integrated in the FastGPT project.

[0058] S4: vectorize the image knowledge in the cultural and tourism knowledge base to form an image knowledge base;

[0059] The process of vectorizing image knowledge mainly uses the Towhee vectorization framework, which provides an interface for calling the ResNet50 model. Through this open source framework, this system designs APIs such as "establishing a database", "adding images", "vectorized image retrieval", and "deleting images" for calling. At the same time, in the data structure of the image knowledge base, each image is added with a corresponding text description. During the image retrieval process, not only image data with high similarity can be returned, but also the corresponding text description. The storage of the image knowledge base is implemented by the Milvus database, which is built locally.

[0060] S5: For the image part of the user's question, query and match in the image knowledge base, obtain images related to the question, and return text description information bound to the image;

[0061] The user's question will be divided into two parts: picture and text, and processed separately by the corresponding system. Figure 2 As shown in the left part, the image part of the user's question is input into the image knowledge base retrieval system based on the Towhee framework and the Milvus database. The Towhee framework calls the ResNet50 model to convert the user's image into a vector form. The Milvus database performs a vector-based comparative search on the user's image and the images stored in the database, and finally returns the image ID with a small distance value (i.e., a large similarity). In order not to affect the storage efficiency and retrieval efficiency of Milvus, the MySQL database finally searches for the image ID and returns the corresponding image and the text description information bound to the image.

[0062] S6: Combine the text part of the user's question and the text description information bound to the image obtained in step S5, search for matches in the text knowledge base, and obtain related text data;

[0063] like Figure 2 As shown in the middle part, the text part of the user's question will be combined with the text description information bound to the image obtained in step S5, and input together into the FastGPT text knowledge base processing system. The FastGPT system calls the locally deployed M3E model to convert all text into vector form, and the PostgreSQL database performs a vector-based comparative search on the text. This operation is completed within the FastGPT system, and finally the associated text data results are obtained.

[0064] S7: inputting the text part of the user's question and the associated text data obtained in step S6 into the large language model to generate prompt words of the large language model, and the large language model generates answer content based on the prompt words;

[0065] like Figure 2 As shown in the right part, the text part of the user's question will be input into the FastGPT system again together with the associated text data obtained in step S6. The FastGPT system will also be responsible for calling the large language model. The large language model will enhance its understanding of the user's question by retrieving the associated text data and give more practical answers in the field of culture and tourism.

[0066] S8: Design and deploy a web application that facilitates user access to the system.

[0067] The present invention realizes a web-based user interface through the gradio library in Python. This interface is not only concise and intuitive, easy for users to operate, but also supports two interactive modes: image and text. Through this interface, users can easily upload pictures and input queries, and the system can quickly provide relevant travel information, greatly optimizing the user experience.

Claims

1. A multimodal cultural and tourism knowledge base answering method based on a large language model, characterized in that: The following steps are involved: S1: Build a large language model; S2: Build a cultural and tourism knowledge base containing text and pictures; S3: vectorize the text knowledge in the cultural and tourism knowledge base to form a text knowledge base; S4: vectorize the image knowledge in the cultural and tourism knowledge base to form an image knowledge base; S5: For the image part of the user's question, query and match in the image knowledge base, obtain images related to the question, and return text description information bound to the image; S6: Combine the text part of the user's question and the text description information bound to the image obtained in step S5, search for matches in the text knowledge base, and obtain related text data; S7: Input the text part of the user's question and the associated text data obtained in step S6 into the large language model to generate prompt words for the large language model. The large language model generates answer content based on the prompt words.

2. The multimodal cultural and tourism knowledge base response method based on a large language model according to claim 1 is characterized in that: In step S1, the large language model adopts the ChatGLM model.

3. The multimodal cultural and tourism knowledge base response method based on a large language model according to claim 1 is characterized in that: In step S2, a cultural and tourism knowledge base containing text and pictures is constructed, which specifically includes: Integrate local cultural and tourism knowledge in the form of text and images into a database.

4. The multimodal cultural and tourism knowledge base response method based on a large language model according to claim 1 is characterized in that: In step S3, the text knowledge in the cultural and tourism knowledge base is vectorized, which specifically includes: The text knowledge in the cultural and tourism knowledge base is first converted into vector form using the FastGPT text knowledge base processing system, and then stored in the PostgreSQL database.

5. The multimodal cultural and tourism knowledge base response method based on a large language model according to claim 1 is characterized in that: In step S4, the image knowledge in the cultural and tourism knowledge base is vectorized, which specifically includes: The image knowledge in the cultural and tourism knowledge base is first converted into vector form using the Towhee vectorization framework, and then stored in the Milvus database.

6. A system for implementing the multimodal cultural and tourism knowledge base response method based on a large language model as described in any one of claims 1 to 5, characterized in that: include: Large language model module, used to understand user questions and generate responses; The local knowledge base module stores text and image data related to local culture and tourism; The text vectorization processing module is used to vectorize the text knowledge in the cultural and tourism knowledge base; Image vectorization processing module, used to vectorize image knowledge in the cultural and tourism knowledge base; A query module, used to retrieve text and images in the image knowledge base and the text knowledge base based on user questions; A prompt word generation module is used to generate prompt words to guide the large language model to generate answers; The web application module is used for users to interact with the cultural and tourism model.

Images