User question and answer method and device, equipment, medium and product

Abstract

The invention discloses a user question answering method and device, equipment, a medium and a product, and relates to the field of artificial intelligence, the method comprises the following steps: receiving question information of a user, the question information comprising one or more of text, audio and image; based on the question information, text generation is carried out through a pre-constructed question and answer model to obtain a question and answer text, and the question and answer model is constructed based on a preset multi-modal model, a vector index and a large language model LLM. Therefore, multiple types of user questions such as texts, audios and images are received, then question information is input into a question and answer model obtained by pre-constructing a multi-modal model, a vector index and a large language model LLM for text generation, and a question and answer text is obtained; the problem that in the prior art, a question and answer model cannot understand various types of information such as images, audios and texts, and consequently the answering effect is poor is solved, and the question and answer accuracy of the user is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a user question and answer method, device, equipment, medium and product. Background Art

[0002] Intelligent Question Answering Models (IQAMs) are a class of technologies designed to answer natural language questions asked by users through computer systems. These models leverage progress and integration in multiple key technology fields, enabling them to provide accurate and efficient answers in a wide range of application scenarios.

[0003] However, the existing intelligent question-answering models are unable to understand the non-text content in images, resulting in the loss of a large amount of important non-text information during knowledge retrieval. Non-text content includes: the colors in the image, the movements of the characters, the layout description of the image content, and other types of visual understanding information.

[0004] In addition, during the knowledge retrieval process, information loss often occurs in the fragmented files, making it impossible to understand the entire article or paragraph content structure, resulting in a poor user experience.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of this application is to provide a user question and answer method, apparatus, device, storage medium and computer program product, aiming to solve the technical problem in the prior art that the question and answer model cannot understand various types of information such as images, audio and text, resulting in poor answering results.

[0007] To achieve the above purpose, the present application proposes a user question-answering method, which includes:

[0008] Receiving a user's question information, wherein the question information includes one or more of text, audio, and image;

[0009] Based on the question information, text generation is performed through a pre-constructed question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0010] In one embodiment, before the step of generating text based on the question information by using a pre-built question-answer model to obtain the question-answer text, the method further includes:

[0011] Acquire a pre-training data set, where the pre-training data set includes one or more of an image data set, a text data set, and an audio data set;

[0012] Parsing the pre-training data set through the multimodal model to obtain structured text of the pre-training data set, and constructing a vector index based on the structured text;

[0013] The question-answering model is constructed through the vector index, multimodal model and LLM.

[0014] In one embodiment, the step of parsing the pre-training data set by using the multimodal model to obtain the structured text of the pre-training data set includes:

[0015] Extracting information from the image dataset using the multimodal model to obtain image objects, image scenes, image colors, and image layouts of the image dataset, and summarizing and analyzing the image dataset based on the image objects, image scenes, image colors, and image layouts to obtain structured text of the image dataset; and / or

[0016] The audio data set is converted into text by using the multimodal model to obtain an audio text, and after the audio text is cleaned and standardized, structured text is generated to obtain the structured text of the audio data set.

[0017] In one embodiment, the step of constructing a vector index based on the structured text includes:

[0018] Segmenting the structured text based on a preset format to obtain a segmentation result;

[0019] Generating a text summary through the segmentation result, and modularizing the structured text according to the segmentation result and the text summary to obtain a text system;

[0020] The structured text is vectorized by a vector model to obtain a text vector, and a vector index is constructed based on the text vector and a text system.

[0021] In one embodiment, the step of generating text based on the question information by using a pre-built question-answer model to obtain the question-answer text includes:

[0022] Parsing the question information through the multimodal model to obtain a user question text;

[0023] Performing vector conversion on the user question text to obtain a question text vector;

[0024] Based on the question text vector, perform vector search through the vector index to obtain a search result;

[0025] According to the search results, a question and answer text is generated through the LLM and a preset knowledge base.

[0026] In one embodiment, the step of generating a question-answer text according to the search result by using the LLM and a preset knowledge base includes:

[0027] Evaluate the text segments in the search result by using the LLM to obtain a first evaluation result, wherein the first evaluation result includes the text segments related to the search result;

[0028] Evaluating the image segment in the retrieval result by using the multimodal model to obtain a second evaluation result, wherein the second evaluation result includes a text segment and an image segment related to the image segment in the retrieval result;

[0029] A final evaluation result is output based on the first evaluation result and the second evaluation result, and text is generated through the LLM according to the final evaluation result and the knowledge base to obtain a question and answer text.

[0030] In addition, to achieve the above purpose, the present application also proposes a user question-and-answer device, the device comprising:

[0031] A receiving module, used for receiving user's question information;

[0032] A generation module is used to generate text based on the question information through a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a user question and answer device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the user question and answer method described above.

[0034] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the user question and answer method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the user question and answer method described above are implemented.

[0036] One or more technical solutions proposed in this application have at least the following technical effects:

[0037] The embodiment of the present application proposes a user question and answer method, device, equipment, storage medium and computer program product, which receives the user's question information, wherein the question information includes one or more of text, audio and image; based on the question information, the text is generated by a pre-constructed question and answer model to obtain the question and answer text, wherein the question and answer model is constructed based on a preset multimodal model, vector index and large language model LLM. Thus, multiple types of user questions are received, such as text, audio and image, and then the question information is input into the question and answer model pre-constructed by the multimodal model, vector index and large language model LLM to generate text, and obtain the question and answer text, which solves the problem that the question and answer model in the prior art cannot understand multiple types of information such as images, audio and text, resulting in poor answering effect, and improves the accuracy of user questions and answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0040] Figure 1 A flowchart of the first embodiment of the user question-and-answer method of this application is provided;

[0041] Figure 2 A schematic diagram of information extraction from an image involving the user question-answering method of this application;

[0042] Figure 3 A flowchart diagram of the second embodiment of the user question-and-answer method of this application;

[0043] Figure 4 A schematic diagram of evaluating the question-answering results involved in the user question-answering method of this application;

[0044] Figure 5 A brief flowchart of the user question-and-answer method provided in Example 2 of the present application;

[0045] Figure 6 This is a schematic diagram of the module structure of the user question and answer device according to an embodiment of the present application;

[0046] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the user question and answer method in the embodiment of the present application.

[0047] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0048] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0049] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0050] The main solution of the embodiment of the present application is: obtain a pre-training data set, which includes one or more of an image data set, a text data set and an audio data set; parse the pre-training data set through the multimodal model to obtain the structured text of the pre-training data set, and construct a vector index based on the structured text; construct a question-answering model through the vector index, the multimodal model and the LLM. Extract information from the image data set through the multimodal model to obtain the image objects, image scenes, image colors and image layouts of the image data set, and summarize and analyze the image data set based on the image objects, image scenes, image colors and image layouts to obtain the structured text of the image data set; and / or convert the audio data set into text through the multimodal model to obtain audio text, and generate structured text after cleaning and standardizing the audio text to obtain the structured text of the audio data set. The structured text is segmented based on a preset format to obtain a segmentation result; a text summary is generated based on the segmentation result, and the structured text is modularized based on the segmentation result and the text summary to obtain a text system; the structured text is vectorized through a vector model to obtain a text vector, and a vector index is constructed based on the text vector and the text system. The question information is parsed through the multimodal model to obtain a user question text; the user question text is vectorized to obtain a question text vector; based on the question text vector, a vector search is performed through the vector index to obtain a search result; based on the search result, a question and answer text is generated through the LLM and a preset knowledge base. The text segments in the retrieval results are evaluated by the LLM to obtain a first evaluation result, which includes text segments related to the retrieval results; the picture segments in the retrieval results are evaluated by the multimodal model to obtain a second evaluation result, which includes text segments and picture segments related to the picture segments in the retrieval results; a final evaluation result is output based on the first evaluation result and the second evaluation result, and text is generated by the LLM according to the final evaluation result and the knowledge base to obtain a question-and-answer text. This solves the problem in the prior art that the question-and-answer model cannot understand multiple types of information such as images, audio, and text, resulting in poor answering results, realizes the question-and-answer of users, and improves the accuracy of the user's question-and-answer.Based on the scheme of the present invention, a user question and answer method is designed based on the fact that it is impossible to understand the non-text content in the picture, resulting in the loss of a large amount of important non-text information during knowledge retrieval. The non-text content includes: the color in the picture, the character's action, the layout description of the picture content and other types of visual understanding information, and in the process of knowledge retrieval, information loss often occurs for the fragmented files after segmentation, and the entire article or paragraph content structure cannot be understood, resulting in a poor user experience and low accuracy. The user question and answer method of the present invention is verified to be effective when asking questions to users, and finally the accuracy of user question and answering using the method of the present invention is significantly improved.

[0051] In this embodiment, for the convenience of description, the following description is made by taking the user question and answer device as the execution subject.

[0052] Due to the limitations of natural language processing technology and machine learning algorithms in the existing technology, the accuracy of user questions and answers needs to be improved, especially for the analysis of image, audio and other data. The existing technology cannot understand the non-text content in the image, resulting in the loss of a large amount of important non-text information during knowledge retrieval. In addition, in the process of knowledge retrieval, information loss often occurs for the fragmented files after segmentation, and the structure of the entire article or paragraph cannot be understood. If the user question and answer model does not undergo domain-specific adaptive learning and training, the accuracy will also decrease.

[0053] The present application provides a solution. In user questions and answers, a question and answer model constructed by a multimodal model, a vector index, and a large language model is used to generate text for the user's question information, thereby improving the accuracy of the questions and answers. At the same time, the accuracy and intelligence of the system can be continuously improved through continuous learning and optimization, thereby providing users with better services.

[0054] It can be seen from the above embodiments that the present application receives user question information, wherein the question information includes one or more of text, audio and image; based on the question information, a pre-constructed question-answering model is used to generate text to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM. Thus, multiple types of user questions are received, such as text, audio and image, and then the question information is input into the question-answering model pre-constructed by the multimodal model, vector index and large language model LLM to generate text to obtain a question-answering text, which solves the problem in the prior art that the question-answering model cannot understand multiple types of information such as images, audio and text, resulting in poor answering results, and improves the accuracy of user questions and answers.

[0055] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a user question and answer device, etc. The following takes the user question and answer device as an example to illustrate this embodiment and the following embodiments.

[0056] Based on this, the present application embodiment provides a user question and answer method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the user question and answer method of the present application.

[0057] In this embodiment, the user question and answer method includes steps S01 to S05:

[0058] Step S01, receiving a user's question information, wherein the question information includes one or more of text, audio, and image;

[0059] Before starting to explain this embodiment, it should be clear that intelligent question answering models are a type of technology that aims to answer natural language questions raised by users through computer systems. These models utilize the progress and integration of multiple key technical fields, enabling them to provide accurate and efficient answers in a wide range of application scenarios. However, existing intelligent question answering models cannot understand the non-text content in images, resulting in the loss of a large amount of important non-text information during knowledge retrieval. Non-text content includes: the color in the image, the movements of the characters, the layout description of the image content, and other types of visual understanding information. In addition, during the knowledge retrieval process, information loss often occurs for the fragmented files after segmentation, and the entire article or paragraph content structure cannot be understood, resulting in a poor user experience.

[0060] In this embodiment, the user's question information is received first, wherein the received question information includes but is not limited to text, audio, and image, etc. In addition, in the process of receiving, the user input box mode can be used, or the user can directly input the voice or transmit the image. The input interface of the intelligent question-answering model is usually designed to receive the text, voice or image input by the user. In the case of text input, the user can directly enter the question or command, and the model will parse the text and try to understand the user's intention, and then generate the corresponding answer or perform the corresponding task. For voice input, the interface usually includes a voice recognition function to convert the user's voice into text input, and then hand it over to the model for processing and response. This interface can provide a more natural and direct interactive experience, especially suitable for scenarios that require fast feedback and operation. The combination of the image transmission interface usually involves processing the information extracted from the image and integrating it into the question-answering system. This combination can expand the model's capabilities, enabling it to more accurately answer the user's questions or perform related tasks by analyzing the image content;

[0061] In practical applications, the input interface of the intelligent question-answering model may include the following functions:

[0062] (1) Text input box or voice input button: Users can directly enter questions or commands in the text box, or click the voice input button to start recording and converting it into text;

[0063] (2) Natural Language Processing (NLP) function: The interface needs to have NLP function to understand the natural language input by the user and convert it into a form that can be processed by the computer;

[0064] (3) Real-time input processing: For real-time applications, the interface needs to be able to respond quickly to user input and quickly generate appropriate answers or perform corresponding operations;

[0065] (4) Multi-language support: For international applications, the interface may need to support input and output in multiple languages;

[0066] (5) User Q&A: The interface needs to be able to recognize the user's intent in order to select the correct answer or perform the appropriate action.

[0067] Step S05, based on the question information, text generation is performed through a pre-constructed question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0068] After receiving the user's question information, the text can be generated through the pre-built question-answering model to obtain the corresponding question-answering text, wherein the above-mentioned question-answering model is constructed based on the multimodal model, vector index and large language model LLM;

[0069] Multimodal models refer to artificial intelligence models that can process and integrate data from multiple different modalities. In the fields of computer vision and natural language processing, the development of multimodal models has become increasingly important because information in the real world usually does not only contain text or images, but also combines data from multiple perception methods. Multimodal models are mainly used for (1) processing multiple input types: multimodal models can simultaneously process different input data types such as images, text, and speech. For example, they can accept input of pictures and text descriptions at the same time and obtain richer information from them. (2) integrating information: these models can integrate information from different modalities to provide more comprehensive and accurate understanding and answers. For example, combining images and text for more accurate object recognition or situational understanding. (3) cross-modal learning: multimodal models can learn deeper semantic understanding from data in different modalities through cross-modal learning. This learning method can improve the model's ability to understand complex scenes.

[0070] Vector indexing refers to a method of organizing and querying large-scale vector data using a vector space model. This technology is widely used in fields such as information retrieval, similarity search, and recommendation systems. Its main applications are: (1) Vector representation: Each item or document in the database is usually represented as a vector. This vector can be a representation based on its features, such as term frequency (TF) and inverse document frequency (IDF) weights. (2) Vector space model: After representing documents or items as vectors, the vector space model can be used to measure the similarity between them. Commonly used measurement methods include cosine similarity. (3) Index structure: For fast retrieval and query, an index structure is usually constructed, such as an inverted index or a tree-based index (such as KD-Tree, Ball Tree, etc.). These index structures can effectively reduce the search space and speed up retrieval.

[0071] The LLM (Large Language Model) model usually refers to a large language model. These models are based on deep learning technology, especially the Transformer architecture, and are used to process and generate natural language text. The structure and function of LLM are roughly as follows: (1) Transformer architecture: LLM is usually based on the Transformer architecture. This architecture is centered on the self-attention mechanism and can effectively handle long-distance dependencies. It is suitable for processing large-scale text data. (2) Pre-training and fine-tuning: LLM usually learns general language representations through large-scale pre-training, and then can be fine-tuned to adapt to specific tasks or application areas, such as question-answering systems, text generation, etc. (3) Application areas: LLM has a wide range of applications in the field of natural language processing, including language understanding, text classification, sentiment analysis, machine translation, and summary generation.

[0072] Specifically, in step S05, before the step of generating text based on the question information by using a pre-built question-answer model to obtain the question-answer text, the method further includes:

[0073] Step S02, obtaining a pre-training data set, wherein the pre-training data set includes one or more of an image data set, a text data set, and an audio data set;

[0074] Step S03, parsing the pre-training data set through the multimodal model to obtain a structured text of the pre-training data set, and constructing a vector index based on the structured text;

[0075] Step S04, constructing a question-answering model through the vector index, multimodal model and LLM.

[0076] Before the question-answering model generates text, this embodiment also provides a corresponding construction process. It should be clear that in the construction of the question-answering model, corresponding settings should be made according to actual business needs. Due to the differences in fields and application objects, the use of the final model requires pre-training and construction. Based on actual business needs, for example, the current large model is used for an online shopping platform. At this time, the corresponding pre-training data set is obtained, such as product name, product image, product type, and search keywords, etc., and the pre-training data set includes at least the corresponding image data set, text data set, and audio data set;

[0077] The specific use of the multimodal model has been described in the above embodiments, so the multimodal model can be used to parse the pre-training data set to obtain the structured text of the pre-training data set, wherein structured text refers to organizing and representing text data in a specific way so that it is easy to process, analyze and understand. Through structured text, a large amount of natural language data can be made easier to manage, analyze and utilize, providing strong support for data-driven decision-making and applications;

[0078] In order to obtain a more relevant question-and-answer effect to the user's question, this embodiment constructs a vector index based on the obtained structured text. Vector indexing is a technology that converts data into vector representation for efficient retrieval and matching. This method is particularly effective when processing a large amount of unstructured data (such as text, images, audio, etc.);

[0079] The process of generating a vector index based on structured text can be:

[0080] (1) Data preprocessing: First, structured text needs to be preprocessed to convert it into a form suitable for vectorization. Specifically, it includes text cleaning and standardization, removing special characters and punctuation marks, converting to lowercase letters, etc., word segmentation, splitting the text into word or phrase units to form a vocabulary, and stemming or word form restoration, converting words to their basic form to reduce the size of the vocabulary and improve generalization ability.

[0081] (2) Vectorized text: Use deep learning models or traditional word embedding technology to convert text into vector representation. Word embedding models, such as Word2Vec, GloVe, and FastText, can convert each word into a vector of fixed length to capture the semantic and contextual information of the word. Pre-trained language models, such as BERT and GPT, can generate context-related word embeddings to more accurately express the meaning of a word in a given context.

[0082] (3) Building vector index: Use vector index libraries (such as FAISS, Annoy, Elasticsearch, etc.) to build indexes, create index structures, select appropriate index structures (such as inverted index, kd tree, LSH, etc.) to efficiently store and retrieve vectors, build indexes, load preprocessed and vectorized text data into the index, and map each document or text paragraph to its corresponding vector;

[0083] After the vector index is constructed, a specific question-answering model can be constructed through the vector index, multimodal model and LLM.

[0084] More specifically, the above step S03, the step of parsing the pre-training data set by the multimodal model to obtain the structured text of the pre-training data set includes:

[0085] Step S031, extracting information from the image dataset using the multimodal model to obtain image objects, image scenes, image colors, and image layouts of the image dataset, and summarizing and analyzing the image dataset based on the image objects, image scenes, image colors, and image layouts to obtain structured text of the image dataset; and / or

[0086] Step S032: convert the audio data set into text using the multimodal model to obtain an audio text, and after cleaning and standardizing the audio text, generate a structured text to obtain a structured text of the audio data set.

[0087] This embodiment mainly describes the structured text conversion of image datasets and audio datasets, which can be specifically as follows:

[0088] For the image dataset, this embodiment extracts information through a multimodal model to obtain the image objects, image scenes, image colors, and image layouts of the image dataset, such as Figure 2 As shown, if the existing technical solution is adopted, the extraction result is as follows: "Mother and Baby Experience Hall Cooling Moisturizing Boneless and Traceless Gentle and Breathable "Ucosi" Set" It can be seen that based on the existing solution, the extraction of image content can only extract the text information in it, and the non-text content cannot be understood, which greatly affects the accuracy of subsequent knowledge retrieval and answering;

[0089] In this embodiment, for Figure 2 Information extraction is performed to obtain "the picture shows a little girl wearing pink pajamas, standing in front of a white background, her hair is tied into two braids and held in her hands, there is a text "POP Mother and Baby Experience Hall" in the upper right corner of the picture, and in the lower part of the picture, there are descriptions of the products: "Cool and moisturizing", "Boneless and traceless" and "Soft and breathable "Ukosi" suit". It can be seen that in this embodiment, the extraction of the content of the picture can well understand the characters, roles, age, clothing, actions and other forms in the picture. In addition, the layout and orientation of the text are very clear and accurate, which can greatly improve the knowledge of knowledge retrieval and answering, and provide more accurate and effective information to the large model, and this information can be used for the final question and answer. For example, without considering the age issue, only clothes can be reflected, and information features such as children's clothes cannot be reflected. In this embodiment, multiple features are obtained, such as "ponytail" and "character image" can correspond to children, and "pink" can correspond to girls, so more accurate results can be obtained;

[0090] For the audio data set, the specific processing flow of this embodiment is as follows:

[0091] (1) Data preparation and preprocessing, including audio datasets: collecting audio datasets containing speech content and audio preprocessing: preprocessing the audio, such as removing noise and reducing the audio level, to improve the quality and accuracy of subsequent processing;

[0092] (2) Multimodal model selection: Select a multimodal model suitable for processing audio data and text generation. For example, select a model that includes speech recognition (ASR) and text generation. Combine speech recognition engines (such as Google Speech-to-Text, Microsoft Azure Speech Service, etc.) and text generation models (such as GPT-4, BERT, etc.). End-to-end multimodal models, such as directly inputting audio data and text data into one model to process and generate them in a joint learning manner.

[0093] (3) In this embodiment, the audio may be first converted to text, for example, using a selected speech recognition engine to convert the audio data into a preliminary text output, and then the preliminary text is processed, including error correction, entity recognition, grammar checking, etc., to improve the quality and structure of the text, and then a multimodal model is input, the processed audio data and text are input into the multimodal model, and finally a structured text is output. It should be clear that the result of the model output may be a structured text, which may be a sentence, paragraph or larger text unit, depending on the design of the model and the task requirements;

[0094] (4) Model selection and adjustment, that is, selecting a suitable multimodal model according to specific application scenarios and requirements, and adjusting and optimizing it to improve the performance and output quality of the model.

[0095] Furthermore, in the above step S03, the step of constructing a vector index based on the structured text includes:

[0096] Step S033, segmenting the structured text based on a preset format to obtain a segmentation result;

[0097] Step S034, generating a text summary through the segmentation result, and modularizing the structured text according to the segmentation result and the text summary to obtain a text system;

[0098] Step S035, performing vector conversion on the structured text through a vector model to obtain a text vector, and constructing a vector index based on the text vector and the text system.

[0099] The vector index construction process in this embodiment is as follows:

[0100] (1) Further subdividing structured text into smaller text segments to improve the accuracy of retrieval and matching, which usually includes sentence segmentation (i.e., segmenting text into sentences) and segment segmentation (i.e., segmenting long text into smaller segments, such as paragraphs or fixed-length windows);

[0101] (2) Text modularization: The segmented text segments are modularized and prepared for conversion into vectors, including standardization and unified text format, such as case conversion and stop word removal.

[0102] (3) Text conversion: Convert text fragments into vector representations for easy storage and retrieval. This step includes: vectorization model, using a pre-trained vector model (such as m3e-base, etc.) to convert text into vectors, vector representation, and obtain the vector representation of the text for subsequent retrieval and matching;

[0103] (4) Vector storage: The generated text vectors are stored in a vector database to support efficient similarity calculation and retrieval, which usually includes storage structure, selecting an appropriate storage structure (such as HNSW, etc.) to optimize retrieval performance, and index establishment, indexing the vector data to improve retrieval speed;

[0104] Among them, it should be noted that in this embodiment, the extracted information is segmented according to paragraphs or sentences, and a summary is generated for the information of the entire picture, and the text is vectorized, and the segmented text information and summary information are converted into vector form for efficient information retrieval and matching. In the entire question and answer model, due to the large amount of picture information, the corresponding summary and text system are also set up for analysis in this embodiment. The summary includes a lot of information about the picture, such as the object, color, function, etc. of the picture, and the system connects the above-mentioned objects, colors, functions, etc. in series to generate multiple question and answer templates, which can provide different question and answer effects for different user question and answer needs.

[0105] This embodiment adopts the above scheme, specifically by receiving the user's question information, the question information includes one or more of text, audio and image; based on the question information, the text is generated by the pre-built question-answering model to obtain the question-answering text, wherein the question-answering model is constructed based on the preset multimodal model, vector index and large language model LLM. Thus, multiple types of user questions are received, such as text, audio and image, and then the question information is input into the question-answering model pre-built by the multimodal model, vector index and large language model LLM to generate text, and obtain the question-answering text, which solves the problem that the question-answering model in the prior art cannot understand multiple types of information such as images, audio and text, resulting in poor answering effect, and improves the accuracy of user questions and answers.

[0106] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 In step S05, based on the question information, a text is generated by using a pre-built question-answering model to obtain a question-answering text. The user question-answering method further includes steps S051 to S054:

[0107] Step S051, parsing the question information through the multimodal model to obtain the user question text;

[0108] Step S052, performing vector conversion on the user question text to obtain a question text vector;

[0109] Step S053, based on the question text vector, perform vector search through the vector index to obtain a search result;

[0110] Step S054, generating a question and answer text according to the search result through the LLM and a preset knowledge base.

[0111] After the question-answering model is built, it can respond to actual user questions, specifically:

[0112] After receiving the user's question information (including audio, image and text), the question information is parsed through the multimodal model in the question-answering model to obtain the user's question text corresponding to the question information. For example, the user's question is: "picture" and the text "Is this product suitable for me?" Then the multimodal model parses it as follows: parse the picture and obtain the product in the picture "red top", "the wearer is a woman around 20 years old", "the text in the picture includes the brand, size and model's figure" and the text content can be parsed into "product", "suitable" and "me", then it can be clearly known that the user wants to ask whether the product in the picture is suitable for the user;

[0113] Then, the user's question text is converted into a vector to obtain the question text vector. Since the vector index has been pre-built, the vector index is used to perform vector search to obtain specific search results, including the text system corresponding to this type of question and answer and image type.

[0114] Then, the search results and the user's questions are given to the LLM. Here, the search results are highly relevant to the user's questions, so this type of search results can be used as reference information for the user's questions. This type of reference information is given to the large model for summary and answering, and finally the question and answer text is obtained;

[0115] The knowledge base generally refers to a data set or system that stores questions and corresponding answers. These knowledge bases are used to train and optimize the question-answering model so that it can effectively answer questions raised by users. The knowledge base in this embodiment should be clear that:

[0116] (1) Data sources of knowledge base: Knowledge base can be obtained from multiple sources, including but not limited to manually sorted question and answer pairs, information extracted from existing documents, web pages or other materials, open domain question and answer datasets, such as social media, forums, and expert knowledge in professional fields. In addition, it can also include single question and single answer (such as FAQ), multiple questions and multiple answers (such as open domain question and answer), structured data (such as query answers in databases), and unstructured text (such as natural language text).

[0117] (2) Management and organization of the knowledge base: The management tool in this embodiment uses a database management system (DBMS) or a special knowledge management tool to organize and store the knowledge base, classify, annotate and index the knowledge for quick retrieval and use, and also includes updating and maintaining it. The knowledge base is updated and maintained regularly to ensure that the information therein is consistent with reality;

[0118] (3) To train and optimize the question-answering model: This embodiment also includes preprocessing the data in the knowledge base, such as word segmentation, removal of stop words, etc., using machine learning or deep learning technology to train the question-answering model so that it can understand and answer relevant questions, as well as evaluate and tune, evaluate the performance of the model through a test set, and adjust and optimize based on feedback;

[0119] (4) Application areas of knowledge base: intelligent assistants, such as virtual assistants (such as Siri, Alexa, etc.), customer service robots for automatically responding to customer inquiries, education and training, providing learning resources and question-answering services, and information retrieval to help users find information in specific fields, etc.

[0120] Specifically, the step S054, based on the search result, generating a question and answer text through the LLM and a preset knowledge base includes:

[0121] Step S0541, evaluating the text segments in the search result by using the LLM to obtain a first evaluation result, wherein the first evaluation result includes text segments related to the search result;

[0122] Step S0542, evaluating the image segment in the search result by using the multimodal model to obtain a second evaluation result, wherein the second evaluation result includes a text segment and an image segment related to the image segment in the search result;

[0123] Step S0543: output a final evaluation result based on the first evaluation result and the second evaluation result, and generate text through the LLM according to the final evaluation result and the knowledge base to obtain a question and answer text.

[0124] In order to obtain more accurate question and answer text, this embodiment uses LLM and multimodal models to evaluate the retrieval results and then output the question and answer text, as shown in the following example: Figure 4 As shown:

[0125] The text fragments in the retrieval results are evaluated by LLM, and the relevance of the answer to the question is evaluated based on the retrieved text fragments using a large language model (LLM). LLM determines whether the text fragment can effectively answer the user's question through its ability to understand natural language. It should be clear that the first evaluation result in this embodiment at least includes text fragments related to the retrieval results, but in actual use, in order to more accurately answer user questions, picture fragments can also be generated based on the text fragments. For example, for "dark clothes", "dark" represents more content, so corresponding pictures can be generated, such as pictures of red, brown and black clothes, which is more conducive to the accuracy of question and answer.

[0126] The image fragments in the retrieval results are evaluated through a multimodal model. Based on the retrieved image content fragments, a large multimodal model such as Cogv2 is used to evaluate the relevance of the answer to the question. The multimodal model determines whether the image fragment can effectively answer the user's question by understanding the content and text information of the image. The second evaluation result mainly targets the image fragments in the retrieval results, that is, after the evaluation is completed, the corresponding text fragments and image fragments generated are more relevant to the user's question.

[0127] It should be clear that in the above-mentioned embodiment, high-quality text suitable for text generation using the LLM model can be obtained through the first evaluation result and the second evaluation result (obtained after corresponding analysis and interception of the search results). Therefore, the question-and-answer text generated by the LLM model and the knowledge base can more accurately answer user questions.

[0128] Combined with the above user questions, we can get that the user is asking whether a red top is suitable. At this time, the picture clip includes, "Red is suitable for users with fair skin" and "The clothing size is suitable for women with a weight of about 50kg and a height of 160cm", and the text clip only contains the user's question, that is, "Is this product suitable for me?" Therefore, in order to give a rigorous answer at this time, we can first ask the user "Can you give priority to providing information such as the user's body shape, common size, and skin color?" If not, the answer can be, "This product is suitable for customers with a weight of about 50kg, a height of about 160cm, and fair skin. Your information has not been obtained yet, so please refer to the above information, or you can provide personal information and then make a combined judgment."

[0129] If the user provides corresponding information, this embodiment can also make corresponding recommendations based on the user's information. For example, if the user's height and weight do not match, it can be analyzed whether there is a corresponding size in the current product and recommended. Or if the skin color does not match, appropriate recommendations or suggestions will be made based on the information provided by the user. For example, "This red top can be matched with blue jeans, etc., which is more suitable for your daily wear. Because this product is longer, if it is matched with a skirt, it is easy to cause the presentation effect to be not good." Therefore, it should be clear that the question-answering model in this embodiment combines the knowledge base and image information, which can better provide users with evaluation, assessment and recommendation effects.

[0130] This embodiment uses the above scheme to parse the question information through the multimodal model to obtain the user question text; perform vector conversion on the user question text to obtain the question text vector; based on the question text vector, perform vector retrieval through the vector index to obtain the retrieval result; and generate the question and answer text through the LLM and the preset knowledge base according to the retrieval result. Thus, multiple types of user questions are received, such as text, audio, and image, and then the question information is input into the question and answer model pre-constructed by the multimodal model, vector index, and large language model LLM for text generation to obtain the question and answer text, which solves the problem in the prior art that the question and answer model cannot understand multiple types of information such as images, audio, and text, resulting in poor answering results, and improves the accuracy of user questions and answers.

[0131] For example, to help understand the implementation process of the user question-and-answer method obtained by combining the present embodiment with the above-mentioned embodiment 1 and the embodiment, please refer to Figure 5 , Figure 5 A brief flowchart of a user question-and-answer method is provided, specifically:

[0132] Before answering user questions, the obtained data set (in this embodiment, an image data set) needs to be converted into a vector. The data set is mainly converted into structured text through a multimodal model. Then, the structured text is segmented and a text summary is generated, and then the text is modularized. Finally, the text is converted into a vector and stored in the vector, so as to obtain the corresponding vector index.

[0133] In actual usage, after receiving user questions, the multimodal model is also used to extract information and convert structured text from user questions. After the structured text is converted into a vector, vector retrieval is performed through vector indexing to match similar text modules, and then question and answer text is generated through the knowledge base and LLM.

[0134] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the user question and answer method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0135] This application also provides a user question and answer device, please refer to Figure 6 , the user question and answer device comprises:

[0136] The receiving module 10 is used to receive the user's question information;

[0137] The generation module 20 is used to generate text based on the question information through a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is built based on a preset multimodal model, vector index and large language model LLM.

[0138] The user question-and-answer device provided by the present application adopts the user question-and-answer method in the above-mentioned embodiment, which can solve the technical problem that the question-and-answer model in the prior art cannot understand various types of information such as images, audio, and text, resulting in poor answering results. Compared with the prior art, the beneficial effects of the user question-and-answer device provided by the present application are the same as the beneficial effects of the user question-and-answer method provided by the above-mentioned embodiment, and the other technical features in the user question-and-answer device are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0139] The present application provides a user question and answer device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the user question and answer method in the above-mentioned embodiment one.

[0140] Reference below Figure 7, which shows a schematic diagram of the structure of a user question-and-answer device suitable for implementing an embodiment of the present application. The user question-and-answer device in the embodiment of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The user question-and-answer device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0141] like Figure 7 As shown, the user question-answering device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the xxx device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the user question-and-answer device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a user question-and-answer device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0142] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0143] The user question-and-answer device provided by the present application adopts the user question-and-answer method in the above embodiment, which can solve the technical problem that the question-and-answer model in the prior art cannot understand various types of information such as images, audio, and text, resulting in poor answering results. Compared with the prior art, the beneficial effects of the user question-and-answer device provided by the present application are the same as the beneficial effects of the user question-and-answer method provided by the above embodiment, and the other technical features in the user question-and-answer device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0144] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0146] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the user question-and-answer method in the above-mentioned embodiment.

[0147] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0148] The computer-readable storage medium may be included in the user question-and-answer device; or may exist independently without being incorporated into the user question-and-answer device.

[0149] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the user question and answer device, the user question and answer device: receives the user's question information, and the question information includes one or more of text, audio and image; based on the question information, generates text through a pre-built question and answer model to obtain a question and answer text, wherein the question and answer model is constructed based on a preset multimodal model, vector index and large language model LLM.

[0150] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0151] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0152] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0153] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned user question-and-answer method, and can solve the technical problem that the question-and-answer model in the prior art cannot understand various types of information such as images, audio, and text, resulting in poor answering results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the user question-and-answer method provided by the above-mentioned embodiment, and will not be repeated here.

[0154] The present application also provides a computer program product, including a computer program, which implements the steps of the user question and answer method as described above when executed by a processor.

[0155] The computer program product provided by the present application can solve the technical problem that the question-answering model in the prior art cannot understand various types of information such as images, audio, and text, resulting in poor answering results. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the user question-answering method provided by the above embodiment, and will not be elaborated here.

[0156] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A user question and answer method, characterized in that: The method comprises: Receiving a user's question information, wherein the question information includes one or more of text, audio, and image; Based on the question information, text generation is performed through a pre-constructed question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

2. The method according to claim 1, characterized in that Before the step of generating text based on the question information by using a pre-built question-answer model to obtain a question-answer text, the method further includes: Acquire a pre-training data set, where the pre-training data set includes one or more of an image data set, a text data set, and an audio data set; Parsing the pre-training data set through the multimodal model to obtain structured text of the pre-training data set, and constructing a vector index based on the structured text; The question-answering model is constructed through the vector index, multimodal model and LLM.

3. The method according to claim 2, characterized in that The step of parsing the pre-training data set by using the multimodal model to obtain the structured text of the pre-training data set includes: Extracting information from the image dataset using the multimodal model to obtain image objects, image scenes, image colors, and image layouts of the image dataset, and summarizing and analyzing the image dataset based on the image objects, image scenes, image colors, and image layouts to obtain structured text of the image dataset; and / or The audio data set is converted into text by using the multimodal model to obtain an audio text, and after the audio text is cleaned and standardized, structured text is generated to obtain the structured text of the audio data set.

4. The method according to claim 2, characterized in that The step of constructing a vector index based on the structured text comprises: Segmenting the structured text based on a preset format to obtain a segmentation result; Generating a text summary through the segmentation result, and modularizing the structured text according to the segmentation result and the text summary to obtain a text system; The structured text is vectorized by a vector model to obtain a text vector, and a vector index is constructed based on the text vector and a text system.

5. The method according to claim 2, characterized in that The step of generating text based on the question information by using a pre-built question-answer model to obtain a question-answer text comprises: Parsing the question information through the multimodal model to obtain a user question text; Performing vector conversion on the user question text to obtain a question text vector; Based on the question text vector, perform vector search through the vector index to obtain a search result; According to the search results, a question and answer text is generated through the LLM and a preset knowledge base.

6. The method according to claim 5, characterized in that The step of generating a question-answer text according to the search result through the LLM and a preset knowledge base comprises: Evaluate the text segments in the search result by using the LLM to obtain a first evaluation result, wherein the first evaluation result includes the text segments related to the search result; Evaluating the image segment in the retrieval result by using the multimodal model to obtain a second evaluation result, wherein the second evaluation result includes a text segment and an image segment related to the image segment in the retrieval result; A final evaluation result is output based on the first evaluation result and the second evaluation result, and text is generated through the LLM according to the final evaluation result and the knowledge base to obtain a question and answer text.

7. A user question and answer device, characterized in that: The device comprises: A receiving module, used for receiving user's question information; A generation module is used to generate text based on the question information through a pre-built question-answering model to obtain a question-answering text, wherein the question-answering model is constructed based on a preset multimodal model, vector index and large language model LLM.

8. A user question-and-answer device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the user question-and-answer method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the user question and answer method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the user question-and-answer method according to any one of claims 1 to 6 are implemented.

Images