Trout sentiment analysis response method and system based on multi-modal fusion and incremental learning

Abstract

The invention discloses a text travel sentiment analysis response method and system based on multi-modal fusion and incremental learning, and the method comprises the steps: obtaining a text, an image, an audio or a video input by a user, carrying out the query in a text and multi-modal knowledge base through employing a multi-modal retrieval technology, and optimizing a retrieval result through combining sentiment analysis, thereby achieving the purpose of improving the user experience. And finally generating a personalized tourism information response. The system integrates a large language model, text and multi-modal knowledge base construction, and a data vectorization processing and sentiment analysis technology, supports multi-modal input and output, can dynamically adjust retrieval and answer contents, and realizes personalized recommendation according to user feedback. The system also has the capabilities of multi-modal data expansion, dynamic knowledge base updating and voice interaction, can significantly improve the response speed and accuracy of tourism information service, is widely applicable to intelligent and personalized tourism information service scenes, and has relatively high innovativeness and practical value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, especially multi-modal data processing and vector search technologies based on large language models. Specifically, it is a method and system for cultural and tourism sentiment analysis and response based on multi-modal fusion and incremental learning, which is used for efficient and personalized information retrieval and intelligent question answering in the cultural and tourism industry. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent question answering systems based on large language models have been widely used in fields such as information retrieval and dialogue generation. However, there are still certain limitations in some aspects of the existing technologies:

[0003] Existing intelligent question answering systems mainly rely on single-modal data, such as only processing text data, and cannot effectively combine multi-modal information such as images, audio, and videos. Application scenarios in the cultural and tourism field need to process various forms of data input by users (such as scenic spot photos, scenic spot videos, voice questions, geographical locations) to provide more comprehensive intelligent responses.

[0004] In the cultural and tourism field, the information update speed is fast, and new data such as scenic spots, activities, and reviews are constantly generated. Traditional static knowledge bases are difficult to meet the user's requirements for real-time and dynamics. At the same time, when dealing with large-scale data, how to ensure the efficient update and retrieval performance of the knowledge base is also an important technical challenge.

[0005] When users ask questions, they usually express emotions (such as happiness, curiosity, or dissatisfaction). Existing systems are difficult to dynamically adjust the tone and content of the answers according to the user's emotions, resulting in insufficient user experience. How to combine sentiment analysis technology to make the answer content more in line with user needs is an urgent problem to be solved by intelligent question answering systems. Summary of the Invention

[0006] The present invention proposes a method and system for cultural and tourism sentiment analysis and response based on multi-modal fusion and incremental learning, aiming to solve the problems of insufficient multi-modal data processing ability, difficult dynamic knowledge update, and lack of emotional and personalized answers in the current cultural and tourism field. By introducing multi-modal fusion, incremental learning, and sentiment analysis technologies, the present invention realizes intelligent, real-time, and personalized cultural and tourism knowledge retrieval and answering, providing users with a more comprehensive, flexible, and emotion-adapted interaction experience.

[0007] In the technical solution, the present invention is realized through the following steps:

[0008] A method for cultural and tourism sentiment analysis and response based on multi-modal fusion and incremental learning, comprising the following steps:

[0009] S1: Build a large language model and perform multi-modal fusion fine-tuning to obtain a fine-tuned large language model;

[0010] S2: Construct a cultural and tourism knowledge base that includes text, images, audio, video, and geographical information;

[0011] S3: Perform vectorization processing and incremental learning processing on the text data in the cultural and tourism knowledge base to form a text knowledge base;

[0012] S4: Generate multi-modal vectors from the image, audio, and video data in the cultural and tourism knowledge base through vectorization processing and incremental learning processing, and store them in the multi-modal knowledge base;

[0013] S5: Based on the question input by the user, perform retrieval in the text knowledge base and the multi-modal knowledge base respectively to obtain text, picture, video, or audio data related to the question;

[0014] S6: Analyze the sentiment tendency of the user input, and adjust the recommendation method, tone, and style of the retrieval results to obtain sentiment-adjusted text data;

[0015] S7: Combine the sentiment-adjusted text data in step S6 and the relevant information obtained from the multi-modal knowledge base to generate prompt words in the fine-tuned large language model;

[0016] S8: The fine-tuned large language model generates multi-modal response content according to the prompt words;

[0017] S9: Make dynamic personalized tourism scenario or activity recommendations based on the multi-modal response content.

[0018] Further preferably, it includes the following steps:

[0019] S1: Build a large language model and perform multi-modal fusion fine-tuning to obtain a fine-tuned large language model;

[0020] S2: Construct a cultural and tourism knowledge base that includes text, images, audio, video, and geographical information, and through an incremental learning mechanism, support the dynamic update of the database content based on real-time changing cultural and tourism data.

[0021] S3: Perform vectorization processing and incremental learning processing on the text data in the cultural and tourism knowledge base to form a text knowledge base, and at the same time support real-time data update and automatically adjust the content of the text library

[0022] S4: Perform vectorization and incremental learning processing on the multi-modal data in the cultural and tourism knowledge base to form a multi-modal knowledge base, and at the same time support real-time data update and automatically adjust the content of the multi-modal database to be synchronized with the content of the text library.

[0023] S5: Based on the questions input by the user, retrieve in the text knowledge base and the multimodal knowledge base respectively to obtain text, pictures, videos or audio data related to the questions, and use user behavior analysis technology and personalized recommendation algorithms to dynamically adjust the answer content, tone and manner according to the user's historical behavior.

[0024] S6: Analyze the emotional tendency, historical query records and historical answer records of the user input, adjust the recommendation method, tone and style of the retrieval results, and obtain the text data after emotion adjustment;

[0025] S7: Combine the text data after emotion adjustment in step S6 and the relevant information obtained from the multimodal knowledge base to generate optimized prompt words, and generate personalized prompt words in the fine-tuned large language model;

[0026] S8: The fine-tuned large language model generates multimodal response content according to the optimized prompt words. The response content includes text, pictures, audio, and video, and can adjust the style and recommended content in real time according to the user's emotional feedback;

[0027] S9: Make dynamic personalized tourism scenario or activity recommendations based on the multimodal response content, and combine incremental learning technology to further optimize the recommended content according to real-time cultural and tourism information

[0028] In step S1, the model selects ChatGLM or GPT-3 as the large language model. ChatGLM is developed by Zhipu AI, has high precision, bilingual support and fast inference ability, and can efficiently process multimodal inputs and generate multimodal outputs that meet the user's needs. The ChatGLM model is deployed through a local server, and at the same time, the model is fine-tuned for the multimodal requirements of the cultural and tourism scenario, so that it has stronger cultural and tourism-specific knowledge processing capabilities.

[0029] Step S2 constructs a multimodal knowledge base containing text, pictures, audio, video and geographical information by integrating local culture and tourism-related data. And through the incremental processing mechanism, the real-time update of the text and multimodal databases is realized, specifically including:

[0030] Integrate the local cultural and tourism knowledge of text, pictures, audio, video and geographical information into a multimodal database, and preprocess according to the data types of each modality to obtain the cultural and tourism knowledge base.

[0031] In step S3, vectorization processing and incremental learning processing are performed on the text data in the cultural and tourism knowledge base to form a text knowledge base, specifically including: using the FastGPT text knowledge base processing system to perform vectorization processing on the text data in the cultural and tourism knowledge base to obtain text data in vectorized format, and the vectorized format of text data is dynamically updated and stored in the PostgreSQL database through incremental learning to form a text knowledge base.

[0032] Use the FastGPT text processing system to convert the text data in the cultural and tourism knowledge base into high-dimensional vectors and store them in the PostgreSQL database. The FastGPT system calls the M3E model (Moka Massive Mixed Embedding), which is trained from a dataset of 22 million Chinese sentence pairs and has excellent text matching performance. An incremental learning mechanism is added. When real-time added text data is added to the cultural and tourism database, the system does not retrain but only performs vectorization processing on the new data and inserts it into the existing vector database, providing efficient and real-time data updates.

[0033] In step S4, the image, audio, and video data in the cultural and tourism knowledge base are processed through vectorization and incremental learning to generate multi-modal vectors and stored in the multi-modal knowledge base. Specifically, use the Towhee vectorization framework to perform vectorization processing on the image, video, and audio data in the cultural and tourism knowledge base respectively to obtain the vectorized data of the image, the vectorized data of the video, and the vectorized data of the audio. The vectorized data of the image, the vectorized data of the video, and the vectorized data of the audio form multi-modal vectors. The multi-modal vectors are dynamically updated and stored in the Milvus database through incremental learning to form a multi-modal knowledge base. Use the Towhee framework to vectorize the picture, audio, and video data to generate high-dimensional embedding representations and store them in the Milvus vector database. Specifically, image feature extraction, video feature extraction, and audio feature extraction operations are applied, and the ResNet50, 3D convolutional network, and VGGish models are used respectively for multi-modal feature embedding representations. After introducing the incremental learning mechanism, the Towhee framework only performs vectorization processing on the real-time added multi-modal data to generate corresponding high-dimensional vector representations and dynamically inserts them into the Milvus vector database. Milvus is a database designed specifically for vector retrieval, supporting efficient similarity calculation and dynamic updates. The system will optimize the Milvus database regularly to ensure that the vector representations of new and old data can work together efficiently while maintaining the retrieval performance.

[0034] Step S5 is expressed as after the system receives the question input by the user, first analyze the input content and decompose it into a text part and a multi-modal part (such as pictures, videos, etc.).

[0035] The system combines the user input, retrieval results, and sentiment analysis results to generate an adapted large language model prompt and input it into the ChatGLM model to generate the final multi-modal answer content. For example, when the user uploads a picture and asks "Is this scenic spot worth visiting?", the system will return a text description of the scenic spot, relevant photos, and recommended videos.

[0036] The final output result is interacted with the user by the system through a web application. Through the friendly design of the web interface, it supports various interaction methods such as user text input, voice input, and picture upload. The final answer content is intuitively presented to the user in the form of pictures, videos, audio, etc.

[0037] Through the combination of the above technologies, the present invention realizes an intelligent and personalized cultural and tourism knowledge base answering service. The system can efficiently process complex multi-modal interaction scenarios, support comprehensive retrieval from text to pictures, audio, video, and geographical information, and at the same time dynamically adapt to new data and adjust the answer content according to the user's emotion. This not only significantly improves the intelligent question-answering ability in the cultural and tourism field, but also greatly improves the user's interaction experience by providing personalized recommendations and multi-modal answers.

[0038] Further preferably, a cultural and tourism sentiment analysis answering method based on multi-modal fusion and incremental learning includes:

[0039] Provide a large language model, which is based on a pre-trained large-scale language model and can process and understand natural language input, serving as the core component for understanding user questions and generating answers;

[0040] Construct a local knowledge base containing text, pictures, and audio-visual materials, integrating multi-modal information related to local culture and tourism, including text descriptions and picture data;

[0041] Perform vectorization processing on the text data in the local knowledge base to generate a text vector representation that is convenient for retrieval and matching;

[0042] Perform vectorization processing on the picture data in the local knowledge base to generate an embedded representation of the image to support picture-based retrieval and matching; Based on natural language processing models and image embedding technologies, the vectorization processing of text and pictures can achieve efficient semantic matching and similarity calculation, ensuring the accuracy of system retrieval. Input the generated answer into the knowledge base and submit it to the large model.

[0043] For the picture-related part in the user's question, query the picture knowledge base and return the pictures associated with the user's question and their description information; By calculating the similarity between the user's question and the multi-modal data in the knowledge base, the system can dynamically sort and select the most relevant knowledge as the answer data source.

[0044] Combine the obtained text data, picture descriptions, and user question data to generate a prompt for prompting the large language model to generate an answer.

[0045] The large language model generates an answer content containing text and picture information based on the prompt to ensure that the user gets a comprehensive and accurate response.

[0046] Based on user sentiment analysis, the system can not only adjust the tone of the response content but also recommend personalized cultural and tourism scenarios or activities that match the user's emotional state.

[0047] Through a reasonable layout of the interaction interface, the web application enables users to easily communicate with the large model.

[0048] Preferably, the vectorization processing of text and images is completed based on natural language processing models and image embedding techniques.

[0049] Preferably, the query matching is performed through similarity calculation. The system matches the user's question with the text and image data in the knowledge base, sorts them according to the matching degree, and selects the most relevant knowledge as the target data.

[0050] A cultural and tourism sentiment analysis response system based on multimodal fusion and incremental learning, characterized by including:

[0051] A large language model module for understanding user questions and generating multimodal responses;

[0052] A local multimodal knowledge base module for storing a cultural and tourism knowledge base containing text, images, audio, video, and geographical information;

[0053] A text vectorization processing module for performing vector processing and incremental learning processing on the text data in the cultural and tourism knowledge base;

[0054] A multimodal vectorization processing module for performing vectorization processing and incremental learning processing on images, videos, and audio in the cultural and tourism knowledge base;

[0055] A query module for retrieving relevant data in the cultural and tourism knowledge base based on user input query text or images, audio, video;

[0056] A sentiment analysis module for identifying user sentiment and adjusting the answering style and recommended content;

[0057] A prompt generation module for guiding the large language model to generate appropriate answers according to user input;

[0058] A web application module for providing an interaction interface for users, supporting multiple input methods for querying and returning multimodal responses.

[0059] The described sentiment analysis module includes:

[0060] A sentiment tendency recognition sub-module for recognizing the sentiment in the user input text, voice, or image;

[0061] A sentiment adaptation module for adjusting the recommended style of text, images, videos, or audio according to the user's sentiment tendency.

[0062] Compared with the prior art, it has the following advantages:

[0063] 1. Multi-modal fusion retrieval accuracy: The present invention constructs a multi-modal knowledge base including text, images, audio, video, and geographical information, and uses advanced vectorization processing technology to achieve cross-modal retrieval.

[0064] 2. Incremental learning for efficient update: An incremental learning mechanism is introduced to achieve real-time dynamic update of the knowledge base. For the text knowledge base, the FastGPT system is adopted. When new text data is added, there is no need to retrain the model. Only the incremental data is vectorized and then inserted into the PostgreSQL database. For the multi-modal knowledge base, the Towhee framework is used. The newly added image, audio, and video data are vectorized in real time and dynamically inserted into the Milvus database, significantly shortening the model fine-tuning time from 6 hours in the traditional solution to 18 minutes, ensuring that the system knowledge always keeps up with the rapid changes in the cultural and tourism industry.

[0065] 3. Sentiment analysis and personalized response: The sentiment analysis module includes a sentiment tendency recognition sub-module and a sentiment adaptation module. Using the RoBERTa sentiment recognition model, it can accurately identify the sentiment in the user's input text, speech, or image. Based on this, the system can adjust the answering style accordingly. For example, for users expressing dissatisfaction, an appeasing tone is adopted and improvement solutions are provided.

[0066] 4. Data processing and storage optimization: The M3E model is used for text vectorization processing, which is trained based on a dataset of 22 million Chinese sentence pairs and has excellent semantic matching performance; the ResNet50 model is used for image vectorization, which can effectively extract key features of images. The vectorized text data and image, video, and audio data are stored in the PostgreSQL and Milvus databases respectively. The former supports complex text vector retrieval, and the latter is designed specifically for vector retrieval, ensuring the efficient storage and fast retrieval of multi-modal data and meeting the large-scale data processing requirements in the cultural and tourism field. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flowchart of the method for the large cultural and tourism model in an embodiment of the present invention.

[0068] Figure 2 It is a structural diagram of the system for the large cultural and tourism model in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0069] As Figure 1 shown, the method of the present invention is mainly divided into the following nine steps:

[0070] S1: Build a large language model:

[0071] This invention selects the domestic open-source large language model ChatGLM, which is developed by Zhipu AI, supports bilingual high-precision inference, and has fast response and cross-platform compatibility. To further optimize the system efficiency and reduce the computing cost, this invention introduces the ChatGLM.cpp acceleration technology. By optimizing the model architecture and computing process through the C++ language, the model significantly reduces the demand for hardware resources while maintaining high performance. In addition, this system builds the large language model through Docker containerization deployment to ensure the independence and maintainability of the modules.

[0072] S2: Build a cultural and tourism knowledge base containing text, pictures and multi-modal information:

[0073] The construction of the cultural and tourism knowledge base is divided into two parts: text and pictures, and is further extended to audio and video data. Collect and organize text data covering regional cultural heritage, natural scenic spots, historical backgrounds, etc., such as detailed descriptions of ancient towns like Tangxi Ancient Town and Nanxun Ancient Town around Hangzhou. Preprocess the data, including denoising (removing invalid characters), sentence splitting and semantic extraction, and finally form a text-based knowledge base with more than 100,000 words. Collect pictures related to scenic spots, including natural scenery, characteristic buildings, local cuisine, etc., and convert the picture formats (unify to JPEG format), clean up blurred or redundant pictures, and scale all pictures to a standard size of 1024 pixels in width. Integrate audio and video content for scenic spot explanations, transcode and compress the audio and video data, and store it in the audio knowledge base for subsequent calls.

[0074] S3: Vectorization processing of text knowledge:

[0075] Perform vectorization processing on the text data in the knowledge base through the FastGPT text processing system. The system calls the M3E vector model, which is trained with a 22-million Chinese sentence pair dataset and can efficiently process Chinese semantic matching. The FastGPT system provides a seamless docking interface with the M3E model to convert the text data into high-dimensional semantic vectors. To facilitate system expansion and management, run the FastGPT system through Docker containerization deployment and link it with the ChatGLM model and other parts of the knowledge base through the One-API framework. After text vectorization, store it in the PostgreSQL database, which supports complex vector retrieval and data management for subsequent semantic matching.

[0076] S4: Vectorization processing of pictures and multi-modal data:

[0077] The vectorization of image, audio, and video data mainly relies on the Towhee framework. The system converts image data into high-dimensional vector representations by calling the ResNet50 model, preserving the main features in the image, such as texture, color, structure, etc. A 3D convolutional neural network is used to extract the spatio-temporal features of the video, and the VGGish model is used to generate audio embedding vectors. The vectorized image, audio, and video data are stored in the Milvus vector database, supporting efficient vector retrieval and matching.

[0078] S5: Retrieve and match in the knowledge base based on the user's input question:

[0079] The question input by the user is decomposed into multimodal information such as text and images by the system. The images uploaded by the user are input into the retrieval module based on the Towhee framework and Milvus database. The ResNet50 model is called to vectorize the images, and the similarity is calculated with the image vectors already stored in the database. Finally, the most similar images and their associated text descriptions are returned. The text input of the user is processed by the FastGPT system. After being vectorized by the M3E model, it is matched with the text vectors in the PostgreSQL database to retrieve the most semantically relevant text data.

[0080] S6: Sentiment analysis and data integration:

[0081] Combining the image descriptions obtained in step S5 with the user's question text, the system further performs sentiment analysis and semantic integration on the text content. Using sentiment recognition models such as RoBERTa, the user's emotions (such as positive, negative, or neutral) are identified, and the recommended content and response tone are dynamically adjusted. The image descriptions and text retrieval results are merged to form enhanced semantic input for subsequent prompt generation.

[0082] S7: Generation of large language model prompts and generation of response content:

[0083] The enhanced semantic data generated by the system is input into the FastGPT system, which generates prompts and calls the ChatGLM model to generate multimodal response content. According to the user's question, image descriptions, and retrieved text results, prompts with complete semantics are dynamically generated, such as "The user hopes to understand the features of this scenic spot. Please generate relevant text descriptions and recommended images." The ChatGLM model generates multimodal content including text descriptions, image recommendations, audio explanations, etc. based on the prompts to ensure the comprehensiveness and accuracy of the answers.

[0084] S8: Design and deploy a web application for user interaction:

[0085] To enhance the user experience, the present invention constructs a lightweight web-based interactive interface through the Gradio framework in Python, supporting multiple interaction methods:

[0086] S9: Based on user sentiment analysis and behavior feedback, perform dynamic personalized recommendations

[0087] The system provides personalized recommended content for users according to their historical query behaviors, emotional states, and real-time locations:

[0088] Recommendation mechanism: Prioritize recommending scenic spots or activities related to the user's interests and matching the emotional tendency.

[0089] Dynamic adjustment: Combine the user's current location to recommend nearby popular scenic spots or activities in real time.

[0090] Multimodal input: Users can interact by uploading pictures, entering text, or voice queries.

[0091] Intuitive output: The system displays the results in multimodal forms (text, pictures, videos, etc.) to facilitate users to quickly obtain the required information.

[0092] Dynamic recommendation: Combine the user's behavior and sentiment analysis results to provide personalized recommended content, such as nearby popular scenic spots or activities.

[0093] Through the above nine steps, the present invention realizes an intelligent, personalized, and multimodal fusion method for answering questions in the cultural and tourism knowledge base. The system not only supports comprehensive retrieval of multimodal content from text to pictures, audio, and video, but also can dynamically adjust the answer content according to the user's emotional tendency, significantly improving the intelligent level of information services in the cultural and tourism field and the user experience.

[0094] Such as Figure 2 As shown, the system architecture of the cultural and tourism large model of the present invention mainly includes the following core modules:

[0095] Module 1: Multimodal data processing engine

[0096] Adopt a hierarchical architecture design, and integrate text, image, audio, and video processing units at the bottom layer. Text processing is based on the FastGPT system and the M3E vector model to achieve the Chinese semantic parsing ability of 22 million sentence pairs training; image processing calls the ResNet50 model through the Towhee framework to extract 1024-dimensional visual feature vectors, the video stream uses 3D-CNN to capture spatio-temporal features, and audio data generates acoustic embeddings through the VGGish model. After all data is normalized, it is stored in the PostgreSQL (text vector) and Milvus (multimodal vector) databases respectively, supporting millisecond-level cross-modal retrieval.

[0097] Module 2: Intelligent reasoning center

[0098] Deploy the domestic ChatGLM large model as the core inference engine, and adopt a two-stage optimization strategy: 1) Reconstruct the C++ low-level operators through ChatGLM.cpp to achieve a 47% improvement in the inference speed under FP16 quantization; 2) Build an elastic computing cluster based on Docker + Kubernetes to dynamically allocate GPU resources to cope with traffic peaks. The system integrates the LoRA fine-tuning mechanism, which can continuously optimize the model output accuracy according to the data in the cultural and tourism vertical fields.

[0099] Module Three: Situational Awareness Interaction System

[0100] It includes a three-layer processing chain: 1) The input parsing layer uses multimodal alignment technology to perform spatio-temporal encoding on the text and images / voices uploaded by users; 2) The emotion calculation layer deploys the RoBERTa-wwm-ext model to achieve fine-grained emotion recognition (9 emotion dimensions); 3) The context management module maintains the dialogue state machine and constructs a cross-session memory network through the Attention mechanism. A geofence interface is specially designed to access the user's location data in real time to trigger the LBS service.

[0101] Module Four: Dynamic Recommendation Engine

[0102] Build a hybrid recommendation model that integrates three algorithms: collaborative filtering (user behavior analysis), content matching (knowledge base retrieval), and spatio-temporal prediction (ARIMA model). The recommendation strategy engine supports multi-objective optimization, balances 12 parameters such as scenic spot popularity, user preferences, and traffic accessibility, and dynamically adjusts the weights through reinforcement learning. The recommendation results are packaged by the multimodal generation module and can output interactive cards containing AR navigation guidance and 360° panoramic previews.

[0103] Module Five: Service Delivery Platform

[0104] Adopt a microservice architecture design. The front end builds a low-code interaction interface through Gradio, supporting multi-channel input of voice / text / images. The back-end service chain introduces a Circuit Breaker fuse mechanism to ensure automatic degradation in case of a single-point failure. Deploy an intelligent load balancer that can dynamically adjust the data transmission protocol according to the user's device type (mobile / PC / AR glasses) and maintain a response delay within 200ms even in a weak network environment.

[0105] Module Six: Continuous Learning Framework

[0106] Build a data flywheel closed-loop system. After the user feedback data is processed by differential privacy, it flows into the incremental learning pipeline. Design a dual-channel update mechanism: 1) The knowledge base update channel screens high-value samples through an active learning algorithm; 2) The model update channel adopts the Elastic Weight Consolidation (EWC) technique to achieve parameter iteration while avoiding catastrophic forgetting. The system automatically generates a model health report every week, monitoring 28 indicators such as accuracy and deviation value.

[0107] This system achieves loose coupling between modules through containerized orchestration. Each component communicates through the gRPC interface and supports horizontal scaling. After actual measurement, under the server configuration of 4-core CPU / 16GB memory, it can process more than 200 concurrent requests simultaneously, with the cross-modal retrieval accuracy reaching 91.7%, and the response latency controlled within 800ms, significantly superior to traditional cultural and tourism Q&A systems.

Claims

1. A cultural and tourism sentiment analysis response method based on multi-modal fusion and incremental learning, characterized in that It includes the following steps: S1: Build a large language model and perform multi-modal fusion fine-tuning to obtain a fine-tuned large language model; S2: Construct a cultural and tourism knowledge base that includes text, images, audio, video, and geographical information; S3: Perform vectorization processing and incremental learning processing on the text data in the cultural and tourism knowledge base to form a text knowledge base; S4: Generate multi-modal vectors from the image, audio, and video data in the cultural and tourism knowledge base through vectorization processing and incremental learning processing, and store them in the multi-modal knowledge base; S5: Based on the question input by the user, retrieve in the text knowledge base and the multi-modal knowledge base respectively to obtain text, pictures, videos, or audio data related to the question; S6: Analyze the sentiment tendency of the user input, and adjust the recommendation method, tone, and style of the retrieval results to obtain sentiment-adjusted text data; S7: Combine the sentiment-adjusted text data in step S6 and the relevant information obtained from the multi-modal knowledge base to generate prompt words in the fine-tuned large language model; S8: The fine-tuned large language model generates multi-modal response content according to the prompt words; S9: Perform dynamic personalized tourism scenario or activity recommendations based on the multi-modal response content.

2. The method for cultural and tourism sentiment analysis response based on multimodal fusion and incremental learning according to claim 1, wherein In step S1, the large language model uses ChatGLM or GPT-3.

3. The method for answering cultural and tourism sentiment analysis based on multi-modal fusion and incremental learning according to claim 1, wherein In step S2, to construct a cultural and tourism knowledge base that includes text, images, audio, video, and geographical information, specifically including: Integrate the local cultural and tourism knowledge of text, pictures, audio, video, and geographical information into a multi-modal database, and perform preprocessing according to the data type of each modality to obtain the cultural and tourism knowledge base.

4. The method for answering cultural and tourism sentiment analysis based on multi-modal fusion and incremental learning according to claim 1, wherein, In step S3, to perform vectorization processing and incremental learning processing on the text data in the cultural and tourism knowledge base to form a text knowledge base, specifically including: Use the FastGPT text knowledge base processing system to perform vectorization processing on the text data in the cultural and tourism knowledge base to obtain text data in vectorized format. The text data in vectorized format is dynamically updated through incremental learning and stored in the PostgreSQL database to form a text knowledge base.

5. The method for cultural and tourism sentiment analysis response based on multimodal fusion and incremental learning according to claim 1, wherein In step S4, to generate multi-modal vectors from the image, audio, and video data in the cultural and tourism knowledge base through vectorization processing and incremental learning processing, and store them in the multi-modal knowledge base, specifically including: Use the Towhee vectorization framework to perform vectorization processing on the image, video, and audio data in the cultural and tourism knowledge base respectively to obtain the vectorized data of the image, the vectorized data of the video, and the vectorized data of the audio. The vectorized data of the image, the vectorized data of the video, and the vectorized data of the audio form multi-modal vectors. The multi-modal vectors are dynamically updated through incremental learning and stored in the Milvus database to form a multi-modal knowledge base.

6. A cultural and tourism sentiment analysis response system based on multimodal fusion and incremental learning for implementing the method according to any one of claims 1 to 5, characterized in that, It includes: A large language model module, which is used to understand user questions and generate multi-modal responses; A local multi-modal knowledge base module, which is used to store a cultural and tourism knowledge base that includes text, images, audio, video, and geographical information; A text vectorization processing module, which is used to perform vector processing and incremental learning processing on the text data in the cultural and tourism knowledge base; A multi-modal vectorization processing module, which is used to perform vectorization processing and incremental learning processing on the pictures, videos, and audio in the cultural and tourism knowledge base; Query module, querying text, images, audio, or video based on user input and retrieving relevant data from the cultural and tourism knowledge base; Sentiment analysis module, identifying user sentiment and adjusting the response style and recommended content; Prompt generation module, generating prompts that guide the large language model to generate appropriate responses based on user input; Web application module, providing an interactive interface for users, supporting multiple input methods for querying and returning multimodal responses.

7. The system according to claim 6, wherein The described sentiment analysis module includes: Sentiment tendency recognition sub-module, used to recognize the sentiment in the user input text, voice, or image; Sentiment adaptation module, used to adjust the recommended style of text, pictures, videos, or audio according to the user's sentiment tendency.

Images