User question-answering method, device, equipment, medium, and product

Abstract

The present application discloses a user question-and-answer method, apparatus, device, medium, and product, which relate to the field of artificial intelligence. The method comprises: receiving a user's question information, wherein the question information comprises one or more of text, audio, and image; based on the question information, generating text through a pre-built question-and-answer model to obtain a question-and-answer text, wherein the question-and-answer model is constructed based on a preset multimodal model, vector index, and large language model (LLM). Thus, when receiving multiple types of user questions, such as text, audio, and image, the question information is then input into a question-and-answer model pre-built by a multimodal model, vector index, and large language model (LLM) to generate text and obtain a question-and-answer text. This solves the problem in the prior art that the question-and-answer model cannot understand multiple types of information, such as images, audio, and text, resulting in poor answering results, and improves the accuracy of user questions and answers.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a user question-and-answer method, apparatus, device, medium, and product. Background Art

[0002] Intelligent Question Answering Models (IQAMs) are a class of technologies designed to answer natural language questions posed by users through computer systems. These models leverage the progress and integration of multiple key technology fields, enabling them to provide accurate and efficient answers in a wide range of application scenarios.

[0003] However, existing intelligent question-answering models are unable to understand the non-textual content in images, resulting in the loss of a large amount of important non-textual information during knowledge retrieval. Non-textual content includes: the colors in the image, the movements of the characters, the layout description of the image content, and other types of visual understanding information.

[0004] In addition, during the knowledge retrieval process, information loss often occurs in the fragmented files after segmentation, making it impossible to understand the entire article or paragraph content structure, resulting in a poor user experience.

[0005] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a user question-answering method, apparatus, device, storage medium and computer program product, aiming to solve the technical problem in the prior art that question-answering models are unable to understand various types of information such as images, audio and text, resulting in poor answering results.

[0007] To achieve the above objectives, this application proposes a user question-answering method, which includes:

[0008] Receive a user's question information, wherein the question information includes one or more of text, audio, and image;

[0009] Based on the question information, text is generated through a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0010] In one embodiment, before the step of generating text based on the question information using a pre-built question-answer model to obtain the question-answer text, the method further includes:

[0011] Obtain a pre-training dataset, where the pre-training dataset includes one or more of an image dataset, a text dataset, and an audio dataset;

[0012] Parsing the pre-training dataset using the multimodal model to obtain structured text of the pre-training dataset, and constructing a vector index based on the structured text;

[0013] The question-answering model is constructed through the vector index, multimodal model and LLM.

[0014] In one embodiment, the step of parsing the pre-training dataset using the multimodal model to obtain structured text of the pre-training dataset includes:

[0015] Extracting information from the image dataset using the multimodal model to obtain image objects, image scenes, image colors, and image layouts of the image dataset, and summarizing and analyzing the image dataset based on the image objects, image scenes, image colors, and image layouts to obtain structured text of the image dataset; and / or

[0016] The audio data set is converted into text using the multimodal model to obtain audio text, and after the audio text is cleaned and standardized, structured text is generated to obtain the structured text of the audio data set.

[0017] In one embodiment, the step of constructing a vector index based on the structured text includes:

[0018] Segmenting the structured text based on a preset format to obtain a segmentation result;

[0019] Generating a text summary based on the segmentation results, and modularizing the structured text based on the segmentation results and the text summary to obtain a text system;

[0020] The structured text is vectorized by using a vector model to obtain a text vector, and a vector index is constructed based on the text vector and a text system.

[0021] In one embodiment, the step of generating text based on the question information using a pre-built question-answer model to obtain the question-answer text includes:

[0022] Parsing the question information through the multimodal model to obtain a user question text;

[0023] Performing vector conversion on the user question text to obtain a question text vector;

[0024] Based on the question text vector, perform vector search through the vector index to obtain a search result;

[0025] According to the search results, a question and answer text is generated through the LLM and a preset knowledge base.

[0026] In one embodiment, the step of generating a question-answer text based on the search results using the LLM and a preset knowledge base includes:

[0027] evaluating the text segments in the search results by the LLM to obtain a first evaluation result, wherein the first evaluation result includes text segments related to the search results;

[0028] Evaluating the image segments in the retrieval results using the multimodal model to obtain a second evaluation result, where the second evaluation result includes text segments and image segments related to the image segments in the retrieval results;

[0029] A final evaluation result is output based on the first evaluation result and the second evaluation result, and text is generated through the LLM according to the final evaluation result and the knowledge base to obtain a question and answer text.

[0030] In addition, to achieve the above-mentioned purpose, the present application also proposes a user question-answering device, which includes:

[0031] A receiving module, used for receiving user's question information;

[0032] A generation module is used to generate text based on the question information through a pre-built question-answering model to obtain question-answer text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a user question-and-answer device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the computer program is configured to implement the steps of the user question-and-answer method as described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the user question-and-answer method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the user question and answer method described above.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] The embodiments of the present application propose a user question-and-answer method, apparatus, device, storage medium, and computer program product, which receives user question information, including one or more of text, audio, and image; based on the question information, generates text through a pre-built question-and-answer model to obtain a question-and-answer text, wherein the question-and-answer model is constructed based on a preset multimodal model, vector index, and large language model LLM. Thus, when receiving multiple types of user questions, such as text, audio, and images, the question information is then input into the question-and-answer model pre-built by the multimodal model, vector index, and large language model LLM to generate text, and obtain a question-and-answer text. This solves the problem in the prior art that the question-and-answer model cannot understand multiple types of information such as images, audio, and text, resulting in poor answering results, and improves the accuracy of user questions and answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0040] Figure 1 A flowchart of the first embodiment of the user question-answering method of this application is provided;

[0041] Figure 2 This is a schematic diagram of the user question-answering method of this application involving information extraction from an image;

[0042] Figure 3 A flowchart of the second embodiment of the user question-answering method of this application is provided;

[0043] Figure 4 A schematic diagram of the user question-answering method of this application involving the evaluation of question-answering results;

[0044] Figure 5 A schematic diagram of a simplified flow chart of the user question-and-answer method provided in Example 2 of this application;

[0045] Figure 6 This is a schematic diagram of the module structure of the user question-and-answer device according to an embodiment of the present application;

[0046] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the user question and answer method in the embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The main solution of the embodiment of the present application is: obtaining a pre-training data set, which includes one or more of an image data set, a text data set and an audio data set; parsing the pre-training data set through the multimodal model to obtain the structured text of the pre-training data set, and constructing a vector index based on the structured text; constructing a question-answering model through the vector index, the multimodal model and the LLM. Extracting information from the image data set through the multimodal model to obtain the image objects, image scenes, image colors and image layouts of the image data set, and summarizing and analyzing the image data set based on the image objects, image scenes, image colors and image layouts to obtain the structured text of the image data set; and / or converting the audio data set into text through the multimodal model to obtain audio text, and after cleaning and standardizing the audio text, generating structured text to obtain the structured text of the audio data set. The structured text is segmented based on a preset format to obtain a segmentation result; a text summary is generated based on the segmentation result, and the structured text is modularized based on the segmentation result and the text summary to obtain a text system; the structured text is vectorized using a vector model to obtain a text vector, and a vector index is constructed based on the text vector and the text system. The question information is parsed using the multimodal model to obtain a user question text; the user question text is vectorized to obtain a question text vector; based on the question text vector, a vector search is performed using the vector index to obtain a search result; based on the search result, a question and answer text is generated using the LLM and a preset knowledge base. The text segments in the search results are evaluated using the LLM to obtain a first evaluation result, which includes text segments related to the search results. The image segments in the search results are evaluated using the multimodal model to obtain a second evaluation result, which includes text segments and image segments related to the image segments in the search results. A final evaluation result is output based on the first and second evaluation results, and text is generated using the LLM based on the final evaluation result and the knowledge base to obtain a question-and-answer text. This solves the problem in the prior art that question-and-answer models are unable to understand multiple types of information, such as images, audio, and text, resulting in poor answering results. This allows for user question-and-answer services and improves the accuracy of user question-and-answer services.Based on the solution of the present invention, based on the fact that it is impossible to understand the non-text content in the picture, resulting in the loss of a large amount of important non-text information during knowledge retrieval, such as: the color in the picture, the character's movements, the layout description of the picture content and other types of visual understanding information, and in the knowledge retrieval process, for the fragmented files after segmentation, information loss often occurs, and the entire article or paragraph content structure cannot be understood, resulting in a poor user experience and low accuracy. A user question and answer method is designed, and the effectiveness of the user question and answer method of the present invention is verified when asking and answering users. Finally, the accuracy of user question and answering using the method of the present invention is significantly improved.

[0051] In this embodiment, for ease of description, the following description is made with the user question-and-answer device as the execution subject.

[0052] Due to the limitations of natural language processing technology and machine learning algorithms in existing technologies, the accuracy of user questions and answers needs to be improved, especially for the analysis of image, audio and other data. Existing technologies are unable to understand the non-text content in images, resulting in the loss of a large amount of important non-text information during knowledge retrieval. In addition, during the knowledge retrieval process, information loss often occurs in fragmented files after segmentation, and the structure of the entire article or paragraph cannot be understood. If the user question and answer model does not undergo domain-specific adaptive learning and training, the accuracy will also decrease.

[0053] This application provides a solution that uses a question-and-answer model constructed by a multimodal model, vector index, and large language model to generate text from user question information in user questions and answers, thereby improving the accuracy of questions and answers. At the same time, through continuous learning and optimization, the accuracy and intelligence of the system can be continuously improved to provide users with better services.

[0054] As can be seen from the above embodiments, the present application receives user question information, which includes one or more of text, audio, and images; based on the question information, a pre-built question-answering model is used to generate text to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index, and large language model LLM. Thus, multiple types of user questions are received, such as text, audio, and images, and then the question information is input into the question-answering model pre-built by the multimodal model, vector index, and large language model LLM to generate text to obtain a question-answering text. This solves the problem in the prior art that the question-answering model cannot understand multiple types of information such as images, audio, and text, resulting in poor answering results, and improves the accuracy of user questions and answers.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device or user question-and-answer device capable of implementing the above functions. The user question-and-answer device is used as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the embodiment of the present application provides a user question and answer method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the user question-and-answer method of this application.

[0057] In this embodiment, the user question-answering method includes steps S01 to S05:

[0058] Step S01: receiving a user's question information, wherein the question information includes one or more of text, audio, and image;

[0059] Before we begin to explain this embodiment, it should be clear that Intelligent Question Answering Models are a type of technology designed to answer natural language questions raised by users through computer systems. These models utilize the progress and integration of multiple key technology fields, enabling them to provide accurate and efficient answers in a wide range of application scenarios. However, existing Intelligent Question Answering Models are unable to understand the non-textual content in images, resulting in the loss of a large amount of important non-textual information during knowledge retrieval. Non-textual content includes: the color in the image, the movement of the characters, the layout description of the image content, and other types of visual understanding information. In addition, during the knowledge retrieval process, information loss often occurs for fragmented files after segmentation, and the entire article or paragraph content structure cannot be understood, resulting in a poor user experience.

[0060] In this embodiment, priority is given to receiving the user's question information, wherein the received question information includes but is not limited to text, audio, and images, etc. In addition, during the receiving process, the user input box mode can be used, or the user can directly input the question or transmit the image. The input interface of the intelligent question-answering model is usually designed to receive text, voice, or image input from the user. In the case of text input, the user can directly enter the question or command, and the model will parse the text and try to understand the user's intention, and then generate the corresponding answer or perform the corresponding task. For voice input, the interface usually includes a voice recognition function to convert the user's voice into text input, and then hand it over to the model for processing and response. This interface can provide a more natural and direct interactive experience, especially suitable for scenarios that require fast feedback and operation. The combination of the image transmission interface usually involves processing the information extracted from the image and integrating it into the question-answering system. This combination can expand the model's capabilities, enabling it to more accurately answer the user's questions or perform related tasks by analyzing the image content;

[0061] In practical applications, the input interface of the intelligent question-answering model may include the following functions:

[0062] (1) Text input box or voice input button: Users can directly enter questions or commands in the text box, or click the voice input button to start recording and converting it into text;

[0063] (2) Natural Language Processing (NLP) functionality: The interface needs to have NLP functionality to understand the natural language input by the user and convert it into a form that can be processed by the computer;

[0064] (3) Real-time input processing: For real-time applications, the interface needs to be able to quickly respond to user input and quickly generate appropriate answers or perform corresponding operations;

[0065] (4) Multi-language support: For international applications, the interface may need to support input and output in multiple languages;

[0066] (5) User Q&A: The interface needs to be able to recognize the user's intent in order to select the correct answer or perform the appropriate action.

[0067] Step S05: Based on the question information, text is generated through a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0068] After receiving the user's question information, the pre-built question-answering model can be used to generate text to obtain the corresponding question-answer text. The above-mentioned question-answering model is built based on a multimodal model, vector index and large language model (LLM).

[0069] Multimodal models refer to artificial intelligence models that can process and integrate data from multiple modalities. In the fields of computer vision and natural language processing, the development of multimodal models has become increasingly important because information in the real world usually does not just contain text or images, but rather combines data from multiple perceptual modes. Multimodal models are mainly used for (1) processing multiple input types: multimodal models can simultaneously process different input data types such as images, text, and speech. For example, they can accept input images and text descriptions at the same time and obtain richer information from them. (2) integrating information: these models can integrate information from different modalities to provide more comprehensive and accurate understanding and answers. For example, combining images and text for more accurate object recognition or situational understanding. (3) cross-modal learning: multimodal models can learn deeper semantic understanding from data from different modalities through cross-modal learning. This learning method can improve the model's ability to understand complex scenes.

[0070] Vector indexing refers to a method of organizing and querying large-scale vector data using a vector space model. This technology has been widely used in fields such as information retrieval, similarity search, and recommendation systems. Its main applications are: (1) vector representation: each item or document in the database is usually represented as a vector. This vector can be a representation based on its features, such as term frequency (TF), inverse document frequency (IDF) weight, etc.; (2) vector space model: after representing documents or items as vectors, the vector space model can be used to measure the similarity between them. Commonly used measurement methods include cosine similarity; (3) index structure: for fast retrieval and query, an index structure is usually constructed, such as an inverted index or a tree-based index (such as KD-Tree, Ball Tree, etc.). These index structures can effectively reduce the search space and speed up retrieval;

[0071] The LLM (Large Language Model) model usually refers to a large language model. These models are based on deep learning technology, especially the Transformer architecture, and are used to process and generate natural language text. The structure and function of LLM are roughly as follows: (1) Transformer architecture: LLM is usually based on the Transformer architecture. This architecture is centered on the Self-Attention Mechanism and can effectively handle long-distance dependencies. It is suitable for processing large-scale text data. (2) Pre-training and fine-tuning: LLM usually learns general language representations through large-scale pre-training, and then can be fine-tuned to adapt to specific tasks or application areas, such as question-answering systems, text generation, etc. (3) Application areas: LLM has a wide range of applications in the field of natural language processing, including language understanding, text classification, sentiment analysis, machine translation, and summary generation.

[0072] Specifically, before the step of generating text based on the question information by using a pre-built question-answer model to obtain the question-answer text in step S05, the method further includes:

[0073] Step S02: obtaining a pre-training dataset, wherein the pre-training dataset includes one or more of an image dataset, a text dataset, and an audio dataset;

[0074] Step S03: parsing the pre-training dataset using the multimodal model to obtain a structured text of the pre-training dataset, and constructing a vector index based on the structured text;

[0075] Step S04: construct a question-answering model through the vector index, multimodal model and LLM.

[0076] Before the question-answering model generates text, this embodiment also provides a corresponding construction process. It should be clear that the construction of the question-answering model should be based on actual business needs. Due to the differences in fields and application objects, the use of the final model requires pre-training and construction. Based on actual business needs, for example, the current large model is used for an online shopping platform. At this time, the corresponding pre-training data set is obtained, such as product names, product images, product types, and search keywords, etc. The pre-training data set includes at least the corresponding image data set, text data set, and audio data set.

[0077] The specific uses of the multimodal model have been described in the above embodiments. Therefore, the multimodal model can be used to parse the pre-training dataset to obtain structured text of the pre-training dataset. Structured text refers to organizing and representing text data in a specific way, making it easy to process, analyze, and understand. Through structured text, large amounts of natural language data can be made easier to manage, analyze, and utilize, providing strong support for data-driven decision-making and applications.

[0078] To facilitate the generation of more relevant Q&A results for user questions, this embodiment constructs a vector index based on the structured text. Vector indexing is a technique that converts data into vector representations for efficient retrieval and matching. This method is particularly effective when processing large amounts of unstructured data (such as text, images, and audio).

[0079] The process of generating a vector index based on structured text can be:

[0080] (1) Data preprocessing: First, structured text needs to be preprocessed to convert it into a form suitable for vectorization. Specifically, it includes text cleaning and standardization, removing special characters and punctuation, converting to lowercase letters, etc., word segmentation, breaking the text into word or phrase units to form a vocabulary, and stemming or lemmatization, converting words to their basic form to reduce the size of the vocabulary and improve generalization ability;

[0081] (2) Vectorized text: Use deep learning models or traditional word embedding technology to convert text into vector representations. Word embedding models, such as Word2Vec, GloVe, and FastText, can convert each word into a fixed-length vector to capture the semantic and contextual information of the word. Pre-trained language models, such as BERT and GPT, can generate context-dependent word embeddings to more accurately express the meaning of words in a given context.

[0082] (3) Building a vector index: Use a vector index library (such as FAISS, Annoy, Elasticsearch, etc.) to build an index, create an index structure, and select an appropriate index structure (such as inverted index, kd tree, LSH, etc.) to effectively store and retrieve vectors. After index construction, load the preprocessed and vectorized text data into the index. Each document or text paragraph is mapped to its corresponding vector.

[0083] After the vector index is constructed, a specific question-answering model can be constructed through the vector index, multimodal model, and LLM.

[0084] More specifically, the above step S03, parsing the pre-training dataset using the multimodal model to obtain the structured text of the pre-training dataset, includes:

[0085] Step S031: extracting information from the image dataset using the multimodal model to obtain image objects, image scenes, image colors, and image layouts of the image dataset, and summarizing and analyzing the image dataset based on the image objects, image scenes, image colors, and image layouts to obtain structured text of the image dataset; and / or

[0086] Step S032: convert the audio data set into text using the multimodal model to obtain audio text, and after cleaning and standardizing the audio text, generate structured text to obtain the structured text of the audio data set.

[0087] This embodiment mainly describes the structured text conversion of image datasets and audio datasets, which can be specifically as follows:

[0088] For the image dataset, this embodiment extracts information through a multimodal model to obtain the image objects, image scenes, image colors, and image layout of the image dataset, such as Figure 2 As shown in the figure, if the existing technical solution is used, the extraction result is as follows: "Mother and Baby Experience Hall Cooling Moisturizing Boneless and Traceless Soft and Breathable "Eco-Silk" Set". It can be seen that based on the existing solution, the extraction of image content can only extract the text information in it, and the non-text content cannot be understood, which greatly affects the accuracy of subsequent knowledge retrieval and answering;

[0089] In this embodiment, for Figure 2 Information extraction is performed, and the result is "The picture shows a little girl wearing pink pajamas, standing in front of a white background. Her hair is tied into two braids and held in her hands. There is a text "POP Mother and Baby Experience Center" in the upper right corner of the picture. In the lower part of the picture, there are descriptions of the products: "Cool and Moisturizing", "Boneless and Markless", and "Soft and Breathable "Eco-Silk" Set". It can be seen that in this embodiment, the extraction of image content can well understand the characters, roles, age, clothing, actions, etc. in the picture. In addition, the layout and orientation of the text are very clear and accurate, which can greatly improve the knowledge retrieval and answering, and provide more accurate and effective information to the large model. All of this information can be used for the final question and answer. For example, without considering the age issue, only clothes can be reflected, and information features such as children's clothes cannot be reflected. In this embodiment, multiple features are obtained, such as "ponytail" and "character image" can correspond to children, and "pink" can correspond to girls, which can obtain more accurate results;

[0090] For the audio data set, the specific processing flow of this embodiment is as follows:

[0091] (1) Data preparation and preprocessing, including audio datasets: collecting audio datasets containing speech content and audio preprocessing: preprocessing the audio, such as removing noise and reducing the audio level, to improve the quality and accuracy of subsequent processing;

[0092] (2) Multimodal model selection: Select a multimodal model suitable for processing audio data and text generation. For example, select a model that includes speech recognition (ASR) and text generation, combining a speech recognition engine (such as Google Speech-to-Text, Microsoft Azure Speech Service, etc.) and a text generation model (such as GPT-4, BERT, etc.), or an end-to-end multimodal model, such as directly inputting audio data and text data into one model for processing and generation in a joint learning manner;

[0093] (3) In this embodiment, the audio may be converted into text first, for example, by using a selected speech recognition engine to convert the audio data into a preliminary text output, and then the preliminary text is processed, including error correction, entity recognition, grammar checking, etc., to improve the quality and structure of the text. Then, a multimodal model is input, and the processed audio data and text are input into the multimodal model, and finally a structured text is output. It should be clear that the result of the model output may be a structured text, which may be a sentence, paragraph, or larger text unit, depending on the design of the model and the task requirements;

[0094] (4) Model selection and adjustment, that is, selecting a suitable multimodal model according to specific application scenarios and requirements, and adjusting and optimizing it to improve the performance and output quality of the model.

[0095] Furthermore, the step S03 of constructing a vector index based on the structured text includes:

[0096] Step S033, segmenting the structured text based on a preset format to obtain a segmentation result;

[0097] Step S034, generating a text summary based on the segmentation results, and modularizing the structured text based on the segmentation results and the text summary to obtain a text system;

[0098] Step S035 , performing vector conversion on the structured text through a vector model to obtain a text vector, and constructing a vector index based on the text vector and the text system.

[0099] The vector index construction process in this embodiment is as follows:

[0100] (1) Further subdividing structured text into smaller text segments to improve the accuracy of retrieval and matching, which usually includes sentence segmentation (i.e., dividing the text into sentences) and fragment segmentation (i.e., dividing long text into smaller segments, such as paragraphs or fixed-length windows);

[0101] (2) Text modularization: modularize the segmented text segments and prepare them for conversion into vectors, including standardization and unified text formatting, such as case conversion and stop word removal;

[0102] (3) Text to vector: Convert text fragments into vector representations for easy storage and retrieval. This step includes, vectorization model, using pre-trained vector model (such as m3e-base, etc.) to convert text into vectors, vector representation, and obtain the vector representation of the text for subsequent retrieval and matching;

[0103] (4) Vector storage: The generated text vectors are stored in a vector database to support efficient similarity calculation and retrieval, which usually includes storage structure, selecting an appropriate storage structure (such as HNSW, etc.) to optimize retrieval performance, and index establishment, establishing an index for the vector data to improve retrieval speed;

[0104] Among them, it should be noted that in this embodiment, the extracted information is segmented into paragraphs or sentences, and a summary is generated for the information of the entire picture, and the text is vectorized, and the segmented text information and summary information are converted into vector form for efficient information retrieval and matching. In the entire question-answering model, due to the large amount of picture information, the corresponding summary and text system are also set up for analysis in this embodiment. The summary includes a lot of information about the picture, such as the object, color, function, etc. of the picture, and the system connects the above-mentioned objects, colors, and functions in series to generate multiple question-answering templates, which can provide different question-answering effects for different user question-answering needs.

[0105] This embodiment adopts the above scheme, specifically by receiving user question information, wherein the question information includes one or more of text, audio, and image; based on the question information, a pre-built question-answering model is used to generate text to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index, and large language model LLM. Thus, when receiving multiple types of user questions, such as text, audio, and images, the question information is then input into the question-answering model pre-built by the multimodal model, vector index, and large language model LLM to generate text to obtain a question-answering text, which solves the problem in the prior art that the question-answering model cannot understand multiple types of information such as images, audio, and text, resulting in poor answering results, thereby improving the accuracy of user questions and answers.

[0106] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 3 In step S05, based on the question information, a text is generated by using a pre-built question-answering model to obtain a question-answering text. The user question-answering method further includes steps S051 to S054:

[0107] Step S051, parsing the question information through the multimodal model to obtain the user question text;

[0108] Step S052, performing vector conversion on the user question text to obtain a question text vector;

[0109] Step S053, performing vector search based on the question text vector through the vector index to obtain a search result;

[0110] Step S054: Generate a question and answer text based on the search results through the LLM and a preset knowledge base.

[0111] After the question-answering model is built, it can respond to actual user questions, specifically:

[0112] After receiving the user's question information (including audio, image and text), the multimodal model in the question-answering model parses the question information to obtain the user's question text corresponding to the question information. For example, the user asks: "picture" and the text "Is this product suitable for me?" The multimodal model then parses the picture to obtain the following: the product in the picture is "red top", "the wearer is a woman around 20 years old", and "the text in the picture includes the brand, size and model's figure". The text content can be parsed into "product", "suitable" and "me", which clearly shows that the user wants to ask whether the product in the picture is suitable for the user;

[0113] The user's question text is then converted into a vector to obtain a question text vector. Since a vector index has been pre-built, vector search using the vector index yields specific search results, including the text system corresponding to the question and answer type and the image type.

[0114] Subsequently, the search results and the user's question are fed into the LLM. The search results are highly relevant to the user's question, so these search results can be used as reference information for the user's question. This reference information is then summarized and answered by the large model, ultimately yielding the question-and-answer text.

[0115] A knowledge base generally refers to a data set or system that stores questions and their corresponding answers. These knowledge bases are used to train and optimize question-answering models so that they can effectively answer questions raised by users. The knowledge base in this embodiment should be clear about the following:

[0116] (1) Data sources of the knowledge base: The knowledge base can be obtained from multiple sources, including but not limited to manually sorted question and answer pairs, information extracted from existing documents, web pages or other materials, open domain question answering datasets, such as social media, forums, and expert knowledge in professional fields. In addition, it can also include single question and single answer (such as FAQ), multiple questions and multiple answers (such as open domain question answering), structured data (such as query answers in databases), and unstructured text (such as natural language text).

[0117] (2) Management and organization of the knowledge base: The management tool in this embodiment uses a database management system (DBMS) or a specialized knowledge management tool to organize and store the knowledge base, classify, label, and index the knowledge for quick retrieval and use, and also includes updating and maintaining it. The knowledge base is updated and maintained regularly to ensure that the information therein is consistent with reality;

[0118] (3) To train and optimize the question-answering model: This embodiment also includes preprocessing the data in the knowledge base, such as word segmentation and stop word removal, using machine learning or deep learning technology to train the question-answering model so that it can understand and answer relevant questions, as well as evaluate and tune it. The performance of the model is evaluated through a test set, and adjustments and optimizations are made based on feedback.

[0119] (4) Application areas of knowledge base: intelligent assistants, such as virtual assistants (such as Siri, Alexa, etc.), customer service robots, used to automatically respond to customer inquiries, education and training, providing learning resources and question-answering services, and information retrieval to help users find information in specific fields, etc.

[0120] Specifically, the step S054 of generating a question-answer text based on the search results using the LLM and a preset knowledge base includes:

[0121] Step S0541, evaluating the text segments in the search results by using the LLM to obtain a first evaluation result, wherein the first evaluation result includes text segments related to the search results;

[0122] Step S0542: Evaluate the image segments in the search results using the multimodal model to obtain a second evaluation result, where the second evaluation result includes text segments and image segments related to the image segments in the search results.

[0123] Step S0543: output a final evaluation result based on the first evaluation result and the second evaluation result, and generate text through the LLM according to the final evaluation result and the knowledge base to obtain a question and answer text.

[0124] In order to obtain more accurate question and answer text, this embodiment uses LLM and multimodal model to evaluate the retrieval results and then output the question and answer text, as shown in the following example: Figure 4 As shown:

[0125] The text fragments in the retrieval results are evaluated by LLM, and the relevance of the answer to the question is evaluated based on the retrieved text fragments using a large language model (LLM). The LLM determines whether the text fragment can effectively answer the user's question through its ability to understand natural language. It should be clear that the first evaluation result in this embodiment at least includes text fragments related to the retrieval results, but in actual use, in order to answer user questions more accurately, picture fragments can also be generated based on the text fragments. For example, for "dark clothes", "dark" represents more content, so corresponding pictures can be generated, such as pictures of red, brown and black clothes, which is more conducive to the accuracy of question and answer.

[0126] The image fragments in the retrieval results are evaluated through a multimodal model. Based on the retrieved image content fragments, a large multimodal model such as Cogv2 is used to evaluate the relevance of the answer to the question. The multimodal model determines whether the image fragment can effectively answer the user's question by understanding the content and text information of the image. The second evaluation result mainly targets the image fragments in the retrieval results, that is, after the evaluation is completed, the corresponding text fragments and image fragments are generated to be more relevant to the user's question.

[0127] It should be clear that in the above embodiment, high-quality text suitable for text generation using the LLM model can be obtained through the first evaluation result and the second evaluation result (obtained after corresponding analysis and interception of the retrieval results). Therefore, the question-and-answer text generated by the LLM model and the knowledge base can more accurately answer user questions.

[0128] Combined with the above user question, we can get that the user is asking whether a red top is suitable. At this time, the image clip includes, "Red is suitable for users with fair skin" and "The clothing size is suitable for women weighing about 50 kg and 160 cm tall." The text clip only contains the user's question, that is, "Is this product suitable for me?" Therefore, in order to give a rigorous answer, we can first ask the user "Can you provide information such as the user's body shape, common size, and skin color first?" If not, the answer can be, "This product is suitable for customers with a weight of about 50 kg, a height of about 160 cm, and fair skin. Your information has not been obtained yet, so please refer to the above information, or you can provide personal information and then make a combined judgment."

[0129] If the user provides corresponding information, this embodiment can also make corresponding recommendations based on the user's information. For example, if the user's height and weight do not match, it can be analyzed to see whether there is a corresponding size in the current product and make a recommendation. Or if the skin color does not match, appropriate recommendations or suggestions will be made based on the information provided by the user. For example, "This red top can be paired with blue jeans, etc., which is more suitable for your daily wear. Because this product is longer, if it is paired with a skirt, it is easy to cause the presentation effect to be not good." Therefore, it should be clear that the question-answering model in this embodiment combines the knowledge base and image information, which can better provide users with evaluation, assessment and recommendation effects.

[0130] This embodiment uses the above-mentioned scheme to specifically parse the question information through the multimodal model to obtain the user question text; perform vector conversion on the user question text to obtain a question text vector; based on the question text vector, perform vector retrieval through the vector index to obtain a retrieval result; and generate a question and answer text based on the retrieval result through the LLM and a preset knowledge base. Thus, when receiving multiple types of user questions, such as text, audio, and images, the question information is then input into the question and answer model pre-constructed by the multimodal model, vector index, and large language model (LLM) to generate text and obtain the question and answer text. This solves the problem in the prior art that the question and answer model cannot understand multiple types of information such as images, audio, and text, resulting in poor answering results, and improves the accuracy of user questions and answers.

[0131] For example, to help understand the implementation process of the user question-answering method obtained by combining the above-mentioned embodiment 1 and the embodiment, please refer to Figure 5 , Figure 5 A brief flowchart of a user question-and-answer method is provided, specifically:

[0132] Before answering user questions, the obtained dataset (in this example, an image dataset) needs to be converted into a vector. This is mainly done by using a multimodal model to convert the dataset into structured text. The structured text is then segmented and a text summary is generated before being modularized. Finally, the text is converted into a vector and stored in the vector to obtain the corresponding vector index.

[0133] In actual usage, after receiving user questions, the multimodal model is also used to extract information and convert structured text from user questions. After converting the structured text into vectors, vector retrieval is performed through vector indexing to match similar text modules. Then, question and answer text is generated through the knowledge base and LLM.

[0134] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the user question-and-answer method of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0135] This application also provides a user question and answer device, please refer to Figure 6 , the user question-answering device comprises:

[0136] Receiving module 10, for receiving user's question information;

[0137] The generation module 20 is used to generate text based on the question information through a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0138] The user question-and-answer device provided in this application, which utilizes the user question-and-answer method described in the aforementioned embodiments, can address the technical issue in the prior art where question-and-answer models are unable to understand various types of information, such as images, audio, and text, resulting in poor answering results. Compared to the prior art, the user question-and-answer device provided in this application has the same beneficial effects as the user question-and-answer method described in the aforementioned embodiments. Other technical features of the user question-and-answer device are the same as those disclosed in the aforementioned embodiments and are not further elaborated here.

[0139] The present application provides a user question-and-answer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the user question-and-answer method in the above-mentioned embodiment one.

[0140] Reference below Figure 7, which shows a schematic diagram of the structure of a user question-and-answer device suitable for implementing an embodiment of the present application. The user question-and-answer device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The user question-and-answer device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0141] like Figure 7 As shown, the user question-and-answer device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the xxx device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 can allow the user question-and-answer device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a user question-and-answer device with various systems, it should be understood that it is not required to implement or have all of the systems shown. More or fewer systems may be implemented or have alternatively.

[0142] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0143] The user question-and-answer device provided in this application, which utilizes the user question-and-answer method in the above-described embodiment, can resolve the technical problem in the prior art where question-and-answer models are unable to understand various types of information, such as images, audio, and text, resulting in poor answering results. Compared to the prior art, the user question-and-answer device provided in this application has the same beneficial effects as the user question-and-answer method provided in the above-described embodiment. Other technical features of this user question-and-answer device are the same as those disclosed in the above-described embodiment and are not further elaborated here.

[0144] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0146] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, wherein the computer-readable program instructions are used to execute the user question-and-answer method in the above-mentioned embodiment.

[0147] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0148] The computer-readable storage medium may be included in the user question-answering device; or it may exist independently without being incorporated into the user question-answering device.

[0149] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the user question-and-answer device, the user question-and-answer device: receives the user's question information, and the question information includes one or more of text, audio and image; based on the question information, generates text through a pre-built question-and-answer model to obtain a question-and-answer text, wherein the question-and-answer model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0150] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0151] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0152] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0153] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., a computer program) for executing the above-mentioned user question-and-answer method. This can solve the technical problem in the prior art that question-and-answer models are unable to understand various types of information, such as images, audio, and text, resulting in poor answer results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the user question-and-answer method provided in the above-mentioned embodiment, and will not be repeated here.

[0154] The present application also provides a computer program product, including a computer program, which implements the steps of the user question-and-answer method described above when executed by a processor.

[0155] The computer program product provided in this application can address the technical problem in existing question-answering models that are unable to understand various types of information, such as images, audio, and text, resulting in poor answering results. Compared to existing technologies, the beneficial effects of the computer program product provided in this application are the same as those of the user question-answering method provided in the above-mentioned embodiments, and will not be elaborated on here.

[0156] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A user question-answering method, characterized in that: The method comprises: Receive a user's question information, where the question information includes one or more of text, audio, and image; Based on the question information, text is generated through a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index, and large language model (LLM); Before the step of generating text based on the question information by using a pre-built question-answer model to obtain the question-answer text, the method further includes: Acquire a pre-training dataset, where the pre-training dataset includes one or more of an image dataset, a text dataset, and an audio dataset; Parsing the pre-training dataset using the multimodal model to obtain structured text of the pre-training dataset, and constructing a vector index based on the structured text; The step of constructing a vector index based on the structured text includes: Segmenting the structured text based on a preset format to obtain a segmentation result; Generating a text summary based on the segmentation results, and modularizing the structured text based on the segmentation results and the text summary to obtain a text system; Performing vector conversion on the structured text through a vector model to obtain a text vector, and constructing a vector index based on the text vector and a text system; A question-answering model is obtained by constructing the vector index, the multimodal model, and the LLM; The step of generating text based on the question information by using a pre-built question-answer model to obtain the question-answer text includes: Parsing the question information through the multimodal model to obtain a user question text; Performing vector conversion on the user question text to obtain a question text vector; Based on the question text vector, perform vector search through the vector index to obtain a search result; Generate question and answer texts based on the search results using the LLM and a preset knowledge base; The step of generating a question-answer text based on the search results through the LLM and a preset knowledge base includes: evaluating the text segments in the search results by the LLM to obtain a first evaluation result, wherein the first evaluation result includes text segments related to the search results; Evaluating the image segments in the retrieval results using the multimodal model to obtain a second evaluation result, where the second evaluation result includes text segments and image segments related to the image segments in the retrieval results; A final evaluation result is output based on the first evaluation result and the second evaluation result, and text is generated through the LLM according to the final evaluation result and the knowledge base to obtain a question and answer text.

2. The method according to claim 1, wherein The step of parsing the pre-training dataset using the multimodal model to obtain structured text of the pre-training dataset includes: Extracting information from the image dataset using the multimodal model to obtain image objects, image scenes, image colors, and image layouts of the image dataset, and summarizing and analyzing the image dataset based on the image objects, image scenes, image colors, and image layouts to obtain structured text of the image dataset; and / or The audio data set is converted into text using the multimodal model to obtain audio text, and after the audio text is cleaned and standardized, structured text is generated to obtain the structured text of the audio data set.

3. A user question-answering device, characterized in that: The device comprises: A receiving module, used for receiving user's question information; A generation module is configured to generate text based on the question information using a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index, and large language model (LLM); The generating module is further configured to obtain a pre-training dataset, wherein the pre-training dataset includes one or more of an image dataset, a text dataset, and an audio dataset; Parsing the pre-training dataset using the multimodal model to obtain structured text of the pre-training dataset, and constructing a vector index based on the structured text; Segmenting the structured text based on a preset format to obtain a segmentation result; Generating a text summary based on the segmentation results, and modularizing the structured text based on the segmentation results and the text summary to obtain a text system; Performing vector conversion on the structured text through a vector model to obtain a text vector, and constructing a vector index based on the text vector and a text system; A question-answering model is obtained by constructing the vector index, the multimodal model, and the LLM; Parsing the question information through the multimodal model to obtain a user question text; Performing vector conversion on the user question text to obtain a question text vector; Based on the question text vector, perform vector search through the vector index to obtain a search result; Generate question and answer texts based on the search results using the LLM and a preset knowledge base; evaluating the text segments in the search results by the LLM to obtain a first evaluation result, wherein the first evaluation result includes text segments related to the search results; Evaluating the image segments in the retrieval results using the multimodal model to obtain a second evaluation result, where the second evaluation result includes text segments and image segments related to the image segments in the retrieval results; A final evaluation result is output based on the first evaluation result and the second evaluation result, and text is generated through the LLM according to the final evaluation result and the knowledge base to obtain a question and answer text.

4. A user question-answering device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the user question-and-answer method according to any one of claims 1 to 2.

5. A storage medium, characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the user question-and-answer method according to any one of claims 1 to 2 are implemented.

6. A computer program product, characterized in that The computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the user question-and-answer method according to any one of claims 1 to 2 are implemented.

Images