Multi-modal question answering method and device in customer service application scene

Abstract

The embodiment of the invention provides a multi-mode question answering method and device in a customer service application scene. The multi-modal question answering method in the customer service application scene comprises the steps of obtaining a multi-modal question input by a user, wherein non-image modal data in the multi-modal question is text data and / or audio data; processing the image modal data by adopting a pre-trained visual extraction model to obtain a visual feature vector; processing the visual feature vector by adopting a pre-trained visual semantic mapping model to obtain a mapping semantic vector; obtaining text input data based on the non-image modal data; and inputting the mapping semantic vector and the text input data into a pre-trained large language processing model to obtain an answer result for the multi-modal question. According to the scheme, the method can adapt to the condition that the user inputs the image modal data and the non-image modal data at the same time, so that the obtained answer result is more in line with the intention of the user, namely, the possibility that the answer result is correct is higher.

Description

Technical Field

[0001] The present disclosure relates to the field of deep learning technology, and in particular to a multimodal question answering method and device in a customer service application scenario. Background Art

[0002] Because large language models can better understand the semantics and intentions of user questions, large language models are deployed in question answering systems to improve the accuracy and user experience of question answering systems such as intelligent customer systems. However, existing question answering systems use large language models such as BERT and GPT, which are based on text dialogue and can only process text or audio-to-text data, but cannot process image data.

[0003] In actual applications, users can show photos and videos to the question answering system to explain their questions and needs. Question answering systems that rely solely on text processing cannot meet users' expectations for natural and intuitive interaction. Summary of the invention

[0004] In order to solve the problem that the existing question answering system that simply relies on text processing cannot meet the user's expectations for natural and intuitive interaction, the embodiments of the present disclosure provide a new multimodal question answering method and device in a customer service application scenario.

[0005] In a first aspect, an embodiment of the present disclosure provides a multimodal question answering method in a customer service application scenario, including:

[0006] Acquire a multimodal question input by a user, wherein the multimodal question includes image modality data and non-image modality data, and the non-image modality data is text data and / or audio data;

[0007] Processing the image modality data using a pre-trained visual extraction model to obtain a visual feature vector; and processing the visual feature vector using a pre-trained visual semantic mapping model to obtain a mapping semantic vector;

[0008] Obtaining text input data based on the non-image modality data;

[0009] Inputting the mapped semantic vector and the text input data into a pre-trained large language processing model to obtain an answer result for the multimodal question, and outputting the answer result;

[0010] The visual semantic mapping model is trained using sample data when the parameters of a pre-trained visual extraction model and a pre-trained large language processing model are frozen. The sample data includes multimodal sample question data and answer sample data.

[0011] Optionally, the imaging modality data includes video data;

[0012] The method of using a pre-trained visual extraction model to process the image modality data to obtain a visual feature vector includes:

[0013] Determine key video frames in the video data, and process each of the key video frames using the visual extraction model to obtain extracted visual features of each of the key video frames;

[0014] The corresponding extracted visual features are sorted according to the order of the key video frames to obtain the visual feature vector.

[0015] Optionally, determining a key video frame in the video data includes:

[0016] Using a clustering algorithm to preliminarily extract all video frames in the video data in chronological order to obtain preliminarily screened video frames;

[0017] A histogram comparison analysis is performed on the preliminary screened video frames to determine key video frames in the preliminary screened video frames.

[0018] Optionally, before inputting the mapped semantic vector and the text input data into a pre-trained large language processing model, the method further includes:

[0019] Performing vectorization conversion on the text input data to obtain a search vector; or performing vectorization conversion on the text input data to obtain a conversion vector, and obtaining the search vector based on the conversion vector and a mapping semantic vector;

[0020] Searching a pre-built domain knowledge vector database based on the search vector to obtain recalled knowledge data;

[0021] The step of inputting the mapping semantic vector and the text input data into a pre-trained large language processing model to obtain an answer result for the multimodal question includes: inputting the mapping semantic vector, the text input data and the recalled knowledge data into the large language processing model to obtain an answer result for the multimodal question.

[0022] Optionally, the step of inputting the mapped semantic vector, the text input data, and the recalled knowledge data into the large language processing model to obtain the answer result for the multimodal question includes:

[0023] Writing the mapping semantic vector, the text input data and the recalled knowledge data into the content slot corresponding to the pre-built prompt template to obtain standard input data; the prompt template includes not only the content slot but also the pre-built prompt information;

[0024] The standard input data is input into the large language processing model to obtain the answer result for the multimodal question.

[0025] Optionally, after outputting the answer result to the multimodal question, the method further includes:

[0026] receiving a user's satisfaction rating of the answer result to the multimodal question;

[0027] In a case where the satisfaction score is higher than a satisfaction threshold score, constructing a fine-tuning training sample based on the visual feature vector, the text input data, and the answer result to the multimodal question;

[0028] In the case of freezing the parameters of the large language processing model, fine-tuning training is performed on the visual semantic mapping model based on the fine-tuning training samples to obtain a fine-tuned visual semantic mapping model.

[0029] Optionally, when the satisfaction score is higher than the satisfaction threshold score, the method further includes:

[0030] Determining a fine-tuning difference weight of a corresponding fine-tuning training sample based on the satisfaction score, wherein the fine-tuning difference weight is positively correlated with the satisfaction score;

[0031] The fine-tuning training of the visual semantic mapping model based on the fine-tuning training sample to obtain a fine-tuned visual semantic mapping model includes:

[0032] Processing the visual feature vector with a visual semantic mapping model to obtain a mapping semantic vector;

[0033] Inputting the mapped semantic vector and the text input data into a pre-trained large language processing model to obtain an output probability that the model output is the answer result to the multimodal question;

[0034] Calculating a loss function corresponding to each fine-tuning training sample based on the output probability and the confirmation probability of the answer result to the multimodal question;

[0035] Performing weighted summation based on the fine-tuning difference weights corresponding to each of the fine-tuning training samples and the loss function to obtain a weighted loss function;

[0036] The weighted loss function is used to fine-tune the visual semantic mapping model to obtain a fine-tuned visual semantic mapping model.

[0037] In a second aspect, an embodiment of the present disclosure provides a multimodal question answering device in a customer service application scenario, including:

[0038] An input data acquisition unit, used to acquire a multimodal question input by a user, wherein the multimodal question includes image modality data and non-image modality data, and the non-image modality data is text data and / or audio data;

[0039] An image modality data processing unit, configured to process the image modality data using a pre-trained visual extraction model to obtain a visual feature vector; and process the visual feature vector using a pre-trained visual semantic mapping model to obtain a mapping semantic vector;

[0040] a non-image modality data removing unit, used for processing the non-image modality data to obtain text input data;

[0041] A fusion processing unit, used for inputting the mapped semantic vector and the text input data into a pre-trained large language processing model, obtaining an answer result for the multimodal question and outputting the answer result;

[0042] The visual semantic mapping model is trained using sample data when the parameters of a pre-trained visual extraction model and a pre-trained large language processing model are frozen. The sample data includes multimodal sample question data and answer sample data.

[0043] In a third aspect, an embodiment of the present disclosure provides a computing device, including a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the multimodal question answering method in the customer service application scenario as described above.

[0044] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the storage medium stores a computer program. When the computer program is executed by a processor, the processor implements the multimodal question answering method in the customer service application scenario as described above.

[0045] The method based on the visual processing model-visual semantic mapping model-large language model architecture of the embodiment of the present disclosure is used to convert the image modality data into semantic information that can be understood by the large language model, and then the large prediction model is used to process the aforementioned semantic information and text input data to obtain answers to multimodal questions. It can adapt to the situation where the user inputs both image modality data and non-image modality data at the same time, so that the obtained answer result is more in line with the user's intention, that is, the possibility of the answer result being correct is greater.

[0046] In some embodiments, the visual semantic mapping model is a model obtained by freezing the parameters of the visual processing model and the large language model and training with sample data. Compared with some models that use a single integrated architecture to process imaging modality data and non-imaging modality data, the visual semantic mapping model can be trained by using the already trained visual processing model and the large language model, and only the parameters of the visual processing model and the large language model need to be frozen, so the model training and use costs are relatively low. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art description. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor, including

[0049] Figure 1 is a flow chart of a multimodal question answering method in a customer service application scenario provided by some embodiments of the present disclosure;

[0050] Figure 2 is a flow chart of a visual semantic mapping model training method provided by some embodiments of the present disclosure;

[0051] Figure 3 is a flow chart of a method for fine-tuning a visual semantic mapping model provided by some embodiments of the present disclosure;

[0052] Figure 4 It is a structural diagram of a multimodal question answering device in a customer service application scenario provided by an embodiment of the present disclosure;

[0053] Figure 5 It is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0055] The term "including" and its variations used in this document are open-ended, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description. In this document, relational terms such as "first" and "second" are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that these entities or operations are the same.

[0056] The embodiment of the present disclosure provides a multimodal question answering method in a customer service application scenario, which processes multiple modal data input by a user, uses the multiple modal data to better understand the user's intention, and then outputs a more accurate answer result for the multimodal question. The question answering structure provided by the embodiment of the present disclosure is executed by a computing device.

[0057] Figure 1 is a flow chart of a multimodal question answering method in a customer service application scenario provided by some embodiments of the present disclosure. Figure 1 As shown, the multimodal question answering method in the customer service application scenario provided by the present disclosure includes S110-S150.

[0058] S110: Obtain a multimodal question input by a user.

[0059] The multimodal question input by the user is question data including at least two modal types of data. Specifically, the multimodal data input by the user includes image modality data and non-image modality data. Image modality data can be data that uses images to display information. Non-image modality data can be text data or audio data. The aforementioned text data can be data in text form, which can be natural language text, symbolic language text (such as mathematical symbolic language text), etc.

[0060] In a specific implementation, after the user inputs a multimodal question through an input device, the input device sends the multimodal question to the computing device, so that the computing device obtains the multimodal question input by the user.

[0061] S120: Use a pre-trained visual extraction model to process the image modality data to obtain a visual feature vector.

[0062] After acquiring the image modality data in the multimodal problem, the computing device uses a pre-trained visual extraction model to process the image modality data to obtain a visual feature vector.

[0063] The visual feature vector is obtained by content extraction of the image modality data, and is a vector representing the key information in the image modality data. When the image modality data is image data, the visual feature vector is data representing the key information of the image data. When the image modality data is video data, the visual feature vector is data representing the key information of the video data.

[0064] The visual feature model is a pre-trained model specifically for processing image modality data. In a specific implementation, the visual feature model can be a model of various architectures such as a Swin-Transformer architecture, a ResNet architecture, etc., which is not particularly limited in the embodiments of the present disclosure.

[0065] In some embodiments, the visual feature model is a model specifically used to process image data. It is trained using sample image data and corresponding key information labels. In other embodiments, the visual feature model is a model specifically used to process video data, which is trained using sample videos and corresponding key information labels.

[0066] In a specific implementation, when the image modality data includes video data, the aforementioned S120 may include the following S121-S122.

[0067] S121: Determine key video frames in the video data.

[0068] Video data is data formed by sequentially splicing (that is, sequentially arranging) video frames according to a specific frame frequency. To avoid smearing, adjacent video frames in video data are mostly video frames with very similar content. If all video frames of video data are processed, the amount of processed data will be too large and a large amount of redundant information will be generated. To avoid the aforementioned problems, when processing video data, the computing device first extracts the key video frames in the video data, and then processes the key video frames to reduce the amount of data processing.

[0069] The key video frame is a video frame whose video data can reflect key information. In a specific implementation, after obtaining the video frame data, the computing device can determine the key video frame using the following S1211-S1212.

[0070] S1211: Preliminarily extract all video frames in the video data using a clustering algorithm in chronological order to obtain preliminary screened video frames.

[0071] In a specific implementation, the computing device may use a clustering algorithm to perform cluster analysis on the video data based on the sorting order of each video frame in the video data, with the goal of classifying adjacent video frames with similar content into one category, and determine each clustered frame data set composed of similar adjacent video frames. The computing device may compare the content similarity of adjacent video frames, cluster them into one category when it is determined that the content similarity reaches a specific similarity, and determine whether the similarity of the video frames clustered into one category all reaches the specific similarity, thereby determining each clustered frame data set.

[0072] After determining the clustered frame data sets, the computing device may then determine the preliminary screening video frames from each clustered frame data set. In a specific implementation, the computing device may use a data frame in each clustered frame data set that has the largest average similarity with the remaining other data frames as the preliminary screening video frame.

[0073] S1212: Performing histogram comparison analysis on the preliminarily screened video frames to determine key video frames in the preliminarily screened video frames.

[0074] After obtaining the preliminary screening video frames, the computing device can then perform grayscale analysis on the pixels in each preliminary screening video frame to determine the grayscale histogram of each preliminary screening video frame. After obtaining the grayscale histogram, the video key frame can be determined by combining the histogram comparison of the preliminary screening video frames to remove redundant frames.

[0075] Of course, in practical applications, the method for determining key video frames in video data is not limited to the aforementioned method using clustering and histogram comparison analysis, but may also adopt key frame extraction algorithms used in other existing technologies, such as methods based on image content analysis, methods based on operation analysis, etc.

[0076] S122: Process each key video frame using a visual extraction model to obtain extracted visual features of each key video frame.

[0077] After obtaining the key video frames, the computing device may then use a pre-trained visual extraction model to process each key video frame separately to obtain extracted visual features corresponding to each key video frame.

[0078] In a specific implementation, the visual extraction model using various architectures as exemplified above may be used to process each key video frame to obtain corresponding extracted visual features.

[0079] S123: Sort the corresponding extracted visual features according to the order of the key video frames to obtain a visual feature vector.

[0080] After obtaining the extracted visual features corresponding to each key video frame, the extracted visual features corresponding to each key video frame are spliced ​​in chronological order to obtain a visual feature vector. In the disclosed embodiment, the use of a visual feature vector is only a representation that is easy to understand. In practical applications, the extracted visual features corresponding to each key video frame may be vectors or matrices of the same dimension, and the spliced ​​features formed by combining the extracted visual features may be a two-dimensional tensor or a three-dimensional tensor.

[0081] S130: Using a pre-trained visual semantic mapping model to process the visual feature vector to obtain a mapping semantic vector.

[0082] The visual semantic mapping model is a model that maps the visual feature vector representing the visual modality into a model representing the semantic features. It is a cross-modal model used to bridge the language gap between vision and language models.

[0083] In some embodiments, the visual semantic mapping model can be a model based on the Transformer architecture. For example, it can be a Query Transformer model, which uses the self-attention mechanism provided by the Transformer architecture, uses an encoder to calculate the association between each token and other tokens in the visual feature vector, captures the long-distance dependency in the visual feature vector and performs encoding processing, and then uses a decoder to decode the encoded data into a vector representing a word or subword, and combines the aforementioned vectors to form a mapping semantic vector.

[0084] In a specific implementation, the visual semantic mapping model can be a lightweight mapping model. It is trained using sample data after the visual extraction model and the large language model used later are trained and the parameters are frozen. The aforementioned sample data includes multimodal sample question data and answer sample data. In order to make the solution brief, the training method and fine-tuning method of the visual semantic mapping model are analyzed in the following text.

[0085] The aforementioned S120-S130 are used to process the image modality data, and obtain a mapping semantic vector representing the image modality data. The mapping semantic vector can be directly understood by the subsequent large language model (from another perspective, the mapping semantic vector can be used as an input of the large language model).

[0086] S140: Obtaining text input data based on the non-image modality data.

[0087] As analyzed above, non-image modality data is text data and / or audio data. If the non-image modality data is text data, the text data can be directly used as text input data without conversion. If the non-image modality data is audio data, it needs to be converted into text input data by audio-to-text conversion.

[0088] S150: Input the mapped semantic vector and text input data into a pre-trained large language processing model to obtain an answer result for the multimodal question, and output the answer result.

[0089] After obtaining the mapped semantic vector and text input data, the two data can be combined and processed to obtain model input data for inputting into the pre-trained large language model. The model input data is then input into the pre-trained large language model, and the answer result for the multimodal question can be obtained through the large language model processing.

[0090] The large language model provided in the embodiment of the present disclosure may be a language model based on VisualGLM, etc., and the embodiment of the present disclosure does not specifically limit it. Because the mapping semantic vector is essentially a semantic vector, it can be understood by the large language model, so after the large language processes the mapping semantic vector and the text input data, the answer result can be directly obtained.

[0091] After obtaining the answer result, the computing device can then input the aforementioned answer result. In a specific implementation, the computing device can output the answer result in a text output manner, or can output the answer result in an audio output manner, which is not limited in the embodiment of the present disclosure.

[0092] As analyzed above, after receiving a multimodal question input by a user, the embodiment of the present disclosure first uses a visual processing model to process the image modality data in the multimodal question to obtain a visual feature vector, and then uses a visual semantic mapping model to process the visual feature vector to obtain a mapping semantic vector. Subsequently, the mapping semantic vector and the text input data obtained based on the non-image modality data are simultaneously input into the large language model to obtain an answer result.

[0093] That is to say, the method based on the visual processing model-visual semantic mapping model-large language model architecture of the embodiment of the present disclosure can convert image modality data into semantic information that can be understood by the large language model, and then use the large prediction model to process the aforementioned semantic information and text input data to obtain an answer result. It can adapt to the situation where the user inputs both image modality data and non-image modality data at the same time, so that the obtained answer result is more in line with the user's intention, that is, the possibility of the answer result being correct is greater.

[0094] In addition, as previously analyzed, the visual semantic mapping model used in the embodiment of the present disclosure is a model obtained by freezing the parameters of the visual processing model and the large language model and training with sample data. Compared with the method of using a single integrated model to process image modality data and non-image modality data, the embodiment of the present disclosure can use the already trained visual processing model and the large language model, and only needs to freeze the parameters of the visual processing model and the large language model to train the visual semantic mapping model, and the model training and use costs are relatively low.

[0095] In some embodiments of the present disclosure, in addition to executing the aforementioned S110 - S150 , the computing device may also execute the following S160 - S170 before executing the aforementioned S150 .

[0096] S160: Perform vectorization conversion on the text input data to obtain a retrieval vector.

[0097] S170: Search a pre-built domain knowledge vector database based on the search vector to obtain recalled knowledge data.

[0098] Through the actual use experience of large language models, it is found that providing appropriate reference background knowledge as constraints for large language models can avoid the large language model from generating output hallucinations, and thus avoid outputting obviously wrong answers. In order to achieve the above goals, it is necessary to be able to obtain reference background knowledge for user questions. In the embodiment of the present disclosure, a vector search method is used to search the pre-built domain knowledge vector database to obtain the recalled knowledge data.

[0099] Specifically, as analyzed above, the computing device first performs vectorization conversion on the text input data to obtain a retrieval vector, and then uses the retrieval vector to search the domain knowledge vector database to obtain the recalled knowledge data.

[0100] The aforementioned domain knowledge vector database can be a vector search database constructed using Faiss combined with MySQL, which calculates the semantic similarity between the retrieval vector and the retrieval vector corresponding to each document fragment in the domain knowledge base, and uses the document fragment corresponding to the retrieval vector with the closest semantic similarity and a semantic similarity greater than or equal to a set threshold as the recalled knowledge data.

[0101] The aforementioned domain knowledge vector database is obtained by parsing and splitting the content of the domain knowledge base and creating corresponding retrieval vectors. The document content of the domain knowledge base is parsed and split, and the construction of the domain knowledge vector database includes: parsing and splitting the document content of the domain knowledge base includes: firstly, the original data of the domain document is subjected to layout analysis, text recognition and table parsing operations respectively; then the parsed content is reconstructed through page content reconstruction technology, and the title, text paragraph, table, and illustration information are sorted according to the document content structure, and output in the form of structured information; based on the chapter structure of the outputted entire article, the knowledge text is refined and converted into the standard knowledge base input format, and then converted into vector data through the vector model and stored in the vector database.

[0102] In the specific implementation, the pre-built domain knowledge vector database is searched based on the retrieval vector, and the recalled knowledge data is obtained by calculating the vector similarity between the retrieval vector and the retrieval vector of each document fragment in the domain knowledge base. The aforementioned vector similarity can be represented by cosine similarity or Euclidean distance.

[0103] , vec1=(x1,x2,……,x n ),

[0104] vec2=(y1,y2,……,y n ), the cosine similarity is

[0105] The aforementioned S160 is to perform vectorization conversion on the text input data to obtain a retrieval vector. In other embodiments, the computing device may also perform vectorization conversion on the text input data to obtain a conversion vector, and then obtain a retrieval vector based on the conversion vector and the mapping semantic vector. The aforementioned retrieval vector obtained based on the conversion vector and the mapping semantic vector may be a weighted average of the two, and after obtaining the weighted average vector, the weighted average vector is used as the retrieval vector.

[0106] On the premise that the recalled knowledge data is obtained by executing the aforementioned S160-S170, the aforementioned S150 is specifically the following S151.

[0107] S151: Input the mapped semantic vector, text input data and recalled knowledge data into the large language processing model to obtain the answer result.

[0108] In a specific implementation, in order to enable the large language model to correctly understand the input content, the computing device can write the mapping semantic vector, text input data and recalled knowledge data into the slots corresponding to the pre-built prompt template to obtain standard input data, and then input the standard input data into the large language model to obtain the answer result.

[0109] The prompt template is a template that guides the large language model to generate specific, relevant and diverse outputs by designing input prompts. In the specific implementation, the prompt template includes not only slots for placing the aforementioned mapping semantic vectors, text input data and recalled knowledge data, but also pre-built prompt information. The prompt information in the prompt template includes requirements such as semantic content, text format, and word count that limit the answer results. For example, in a specific application, the prompt template can be as follows: Reference knowledge: {recall knowledge data}; You are a professional and skilled answerer, please patiently ask and answer user questions based on the mapping semantic vectors and reference knowledge; User questions: {text input data}. "

[0110] As mentioned above, the visual semantic mapping model is trained using sample data when the parameters of the pre-trained visual extraction model and the pre-trained large language processing model are frozen. Figure 2 is a flow chart of a visual semantic mapping model training method provided by some embodiments of the present disclosure. Figure 2 As shown, the visual semantic mapping model training method includes S210-S230.

[0111] S210: using a pre-trained visual extraction model to process sample image modality data to obtain a visual feature vector; and using a visual semantic mapping model to be trained to process the visual feature vector to obtain a mapping semantic vector.

[0112] S220: Obtain text input data based on the sample non-image modality data.

[0113] S230: Input the mapping semantic vector and text input data into a pre-trained large language processing model to obtain a model answer result, and calculate a loss function based on the model answer result and the answer sample data, and adjust the model parameters of the visual semantic mapping model based on the loss function.

[0114] In a specific implementation, the computing device processing can repeatedly execute the steps of processing the visual feature vector using the visual semantic mapping model to be trained to obtain the mapping semantic vector, and subsequent steps until the loss function reaches a preset target.

[0115] In practical applications, due to the small number of sample data and other reasons, the visual semantic mapping model may not have good generalization capabilities. However, fine-tuning the model can improve the generalization ability of the visual semantic mapping model. The key to achieving model fine-tuning is to have fine-tuning sample data. Figure 3 FIG. 1 is a flow chart of a method for fine-tuning a visual semantic mapping model provided by some embodiments of the present disclosure. Figure 3 As shown, the fine-tuning method for the visual semantic mapping model includes the following S310-S330.

[0116] S310: Receive the user's satisfaction score for the answer result, and determine whether the satisfaction score is higher than the satisfaction threshold score; if so, execute S320.

[0117] S320: Construct fine-tuning training samples based on the visual feature vector, text input data and answer results.

[0118] In the disclosed embodiment, while sending the answer result to the user, the calculation result can also send a satisfaction rating table to the user at the same time, in order to suggest the user to rate whether the answer structure meets the needs. According to the above analysis, the satisfaction rating is the score of the user's subjective evaluation of whether the corresponding answer result meets the user's expectations. The satisfaction score reflects the quality of the visual extraction model-visual semantic mapping model-large language model in processing characteristic multimodal problems to a certain extent. When the output quality of the visual extraction model and the large language model is relatively reliable, the satisfaction rating is mainly to evaluate the score of the visual semantic mapping model.

[0119] The satisfaction threshold score is pre-set and is used to determine whether the answer result and the corresponding multimodal question can be used to fine-tune the score of the sample data. If the satisfaction score is higher than the satisfaction threshold score, it is determined that the answer result and the corresponding multimodal question can be used as the fine-tuning sample data.

[0120] Here we explain why the visual feature vector and text input data are used as fine-tuning training sample data, but the image modality data and non-image modality data are not used as fine-tuning training data. As analyzed above, because the parameters of the visual extraction model are frozen, the visual feature vector obtained by processing the image modality data using the visual extraction model always remains unchanged, and the text input data obtained based on the non-modal image data also remains unchanged. Therefore, there is no need to repeat the calculation, and the two can be directly used as data in the fine-tuning training sample.

[0121] S330: When the parameters of the large language processing model are frozen, fine-tune the visual semantic mapping model based on the fine-tuning training samples to obtain a fine-tuned visual semantic mapping model.

[0122] When the fine-tuning training sample data is determined, the computing device can freeze the large language processing model parameters, and perform fine-tuning training on the visual semantic mapping model based on the fine-tuning training samples to obtain a fine-tuned visual semantic mapping model.

[0123] It should be noted here that in actual applications, the computing device performs the fine-tuning training process with a large number of fine-tuning training samples to obtain a fine-tuned visual semantic mapping model.

[0124] In a specific implementation, the computing device may adopt the following S331-S332 to implement the training of the visual semantic model.

[0125] S331: Use the visual semantic mapping model to process the visual feature vector to obtain a mapping semantic vector: input the mapping semantic vector and the visual feature vector into the large language model to obtain the output probability corresponding to the aforementioned answer result.

[0126] S332: Calculate a loss function based on the output probability, and fine-tune the visual semantic mapping model based on the loss function to obtain a fine-tuned visual semantic mapping model.

[0127] In a specific implementation, the computing device sets the correct probability of the answer result to 100%, calculates the loss function using the aforementioned 100% correct probability and output probability, and then fine-tunes the visual semantic mapping model based on the loss function to obtain a fine-tuned visual semantic mapping model.

[0128] In some embodiments, the computing device may use methods such as Fingetune, LoRA, QLORA, P-tuning, etc. Since the relevant methods have been documented and there are ready-made program implementation methods, the specific use of the aforementioned fine-tuning training methods will not be analyzed here.

[0129] In the above-mentioned embodiment, when the satisfaction score is higher than the satisfaction threshold score, a training sample is constructed based on the visual feature vector, text input data and answer result. At this time, only the training samples corresponding to different satisfaction scores are given the same weight for training the visual semantic mapping model. However, in actual situations, training samples with higher satisfaction scores have higher credibility, and training samples with lower satisfaction scores have lower credibility. It is not reasonable to give the same weight to the two in training the visual semantic mapping model, which may cause the constructed visual meaning mapping model to have a systematic deviation.

[0130] To solve the problem in the previous paragraph, in an embodiment of the present disclosure, the computing device may further execute the following S340 before executing the aforementioned S330.

[0131] S340: Determine the fine-tuning difference weight of the corresponding fine-tuning training sample based on the satisfaction score.

[0132] The fine-tuning difference weight is a weight that characterizes the importance of the corresponding fine-tuning sample data to the calculation of the subsequent loss function. As analyzed above, if the satisfaction score corresponding to a fine-tuning sample is higher, the probability that it is a true correct sample is greater, and the corresponding fine-tuning difference weight should also be larger and better. In other words, in the embodiment of the present disclosure, the fine-tuning difference weight is positively correlated with the satisfaction score.

[0133] In a specific implementation, the correspondence between each satisfaction score and the fine-tuning difference weight may be pre-selected and determined, and the fine-tuning difference weight of each fine-tuning training sample may be determined through the aforementioned correspondence.

[0134] When executing the aforementioned S340, the aforementioned S332 is specifically the following S3321-S3323.

[0135] S3321: Based on the output probability and the confirmation probability of the answer result, calculate the loss function corresponding to each fine-tuning training sample.

[0136] Specifically, during the execution of S3321, the correct probability of the answer result is set to 100% based on the computing device, and the loss function corresponding to each fine-tuning training sample is calculated using the aforementioned 100% correct probability and output probability.

[0137] S3322: Perform weighted summation based on the fine-tuning difference weights and loss functions corresponding to each fine-tuning training sample to obtain a weighted loss function.

[0138] In the disclosed embodiment, after obtaining the loss function corresponding to each fine-tuning training sample, the computing device does not directly add the loss functions according to the same weight to obtain the final loss function used for model training. Instead, the loss function corresponding to each fine-tuning training sample is weighted according to the fine-tuning difference weight corresponding to the aforementioned fine-tuning training sample to obtain a weighted loss function. As analyzed above, if the fine-tuning difference weight corresponding to the fine-tuning training sample is larger, the corresponding loss function is more important in the weighted loss function, and when the visual semantic mapping model is fine-tuned subsequently, the corresponding fine-tuning training sample has a greater influence on the model parameter fine-tuning results.

[0139] S3323: Fine-tune the visual semantic mapping model using a weighted loss function to obtain a fine-tuned visual semantic mapping model.

[0140] After obtaining the aforementioned fine-tuning results, the computing device can then use the aforementioned fine-tuning training method to train the visual semantic mapping model until the weighted loss function meets the target requirements.

[0141] In addition to providing the aforementioned question answering method, the disclosed embodiment also provides a multimodal question answering device in a customer service application scenario. Figure 4 is a schematic diagram of the structure of a multimodal question answering device in a customer service application scenario provided by an embodiment of the present disclosure. Figure 4 As shown, the multimodal question answering device 400 in the customer service application scenario includes an input data acquisition unit 401, an image modality data processing unit 402, a non-image modality data removal unit 403 and a fusion processing unit 404.

[0142] The input data acquisition unit 401 is used to obtain multimodal questions input by the user, and the multimodal questions include image modal data and non-image modal data, and the non-image modal data is text data and / or audio data; the image modal data processing unit 402 is used to use a pre-trained visual extraction model to process the image modal data to obtain a visual feature vector; the pre-trained visual semantic mapping model is used to process the visual feature vector to obtain a mapping semantic vector; the non-image modal data addition unit 403 is used to process the non-image modal data to obtain text input data; the fusion processing unit 404 is used to input the mapping semantic vector and the text input data into the pre-trained large language processing model to obtain an answer result for the multimodal question and output the answer result; wherein, the visual semantic mapping model is trained using sample data when the parameters of the pre-trained visual extraction model and the pre-trained large language processing model are frozen, and the sample data includes multimodal sample question data and answer sample data.

[0143] In some embodiments, the imaging modality data includes video data; the imaging modality data processing unit 402 determines key video frames in the video data, and uses a visual extraction model to process each key video frame to obtain extracted visual features of each key video frame; the corresponding extracted visual features are sorted according to the order of the key video frames to obtain a visual feature vector.

[0144] In some embodiments, the image modality data processing unit 402 uses a clustering algorithm to perform preliminary extraction of all video frames in the video data in chronological order to obtain preliminary screened video frames; and performs histogram comparison analysis on the preliminary screened video frames to determine key video frames in the preliminary screened video frames.

[0145] In some embodiments, the multimodal question answering device 400 in the customer service application scenario further includes a data retrieval unit. The data retrieval unit first performs vectorization conversion on the text input data to obtain a retrieval vector; or, performs vectorization conversion on the text input data to obtain a conversion vector, and obtains a retrieval vector based on the conversion vector and the mapping semantic vector; then, based on the retrieval vector, a pre-constructed domain knowledge vector database is retrieved to obtain the recalled knowledge data; accordingly, the fusion processing unit 404 inputs the mapping semantic vector, the text input data, and the recalled knowledge data into the large language processing model to obtain the answer result for the multimodal question.

[0146] In some embodiments, the fusion processing unit 404 writes the mapped semantic vector, text input data and recalled knowledge data into the content slot corresponding to the pre-built prompt template to obtain standard input data; the prompt template includes not only the content slot but also pre-built prompt information; then, the standard input data is input into the large language processing model to obtain the answer result.

[0147] In some embodiments, the multimodal question answering device 400 in the customer service application scenario further includes a model fine-tuning unit. The model fine-tuning unit receives the user's satisfaction score for the answer result; when the satisfaction score is higher than the satisfaction threshold score, a fine-tuning training sample is constructed based on the visual feature vector, the text input data, and the answer result; when the parameters of the large language processing model are frozen, the visual semantic mapping model is fine-tuned based on the fine-tuning training sample to obtain a fine-tuned visual semantic mapping model.

[0148] In some embodiments, when the satisfaction score is higher than the satisfaction threshold score, the model fine-tuning unit determines the fine-tuning difference weight of the corresponding fine-tuning training sample based on the satisfaction score, and the fine-tuning difference weight is positively correlated with the satisfaction score. The corresponding fine-tuning training of the visual semantic mapping model based on the fine-tuning training sample to obtain the fine-tuned visual semantic mapping model includes: processing the visual feature vector with the visual semantic mapping model to obtain a mapping semantic vector; inputting the mapping semantic vector and the text input data into the pre-trained large language processing model to obtain the output probability of the model output being the answer result; calculating the loss function corresponding to each fine-tuning training sample based on the output probability and the confirmation probability of the answer result; performing weighted summation based on the fine-tuning difference weight and loss function corresponding to each fine-tuning training sample to obtain a weighted loss function; fine-tuning the visual semantic mapping model using the weighted loss function to obtain the fine-tuned visual semantic mapping model.

[0149] The embodiment of the present disclosure also provides a computing device for implementing the aforementioned method. Figure 5 Schematic diagram of the structure of the computing device provided by the embodiment of the present disclosure. Figure 5 , which shows a schematic diagram of the structure of a computing device 500 suitable for implementing the embodiments of the present disclosure. Figure 5 The computing device shown is only an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0150] like Figure 5 As shown, the computing device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory ROM 502 or a program loaded from a storage device 508 to a random access memory RAM 503. In the RAM 503, various programs and data required for the operation of the computing device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output I / O interface 505 is also connected to the bus 504.

[0151] Typically, the following devices may be connected to the I / O interface 505: input devices 505 including, for example, a touch screen, a touch pad, a camera, a microphone, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the computing device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The computing device 500 having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0152] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0153] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable storage medium, a computer-readable signal medium, or any combination of the above two.

[0154] The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0155] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer readable program codes. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0156] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0157] The computer-readable medium may be included in the computing device, or may exist independently without being incorporated into the computing device.

[0158] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the tester computer, partially on the tester computer, as a separate software package, partially on the tester computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the tester computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0159] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to each embodiment of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system based on hardware that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] The units involved in the embodiments described in the present disclosure may be implemented by software or by hardware. The name of the unit does not, in some cases, constitute a limitation on the unit itself. The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0161] The above are only specific embodiments of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal question answering method in a customer service application scenario, characterized in that: include: Acquire a multimodal question input by a user, wherein the multimodal question includes image modality data and non-image modality data, and the non-image modality data is text data and / or audio data; Processing the image modality data using a pre-trained visual extraction model to obtain a visual feature vector; and processing the visual feature vector using a pre-trained visual semantic mapping model to obtain a mapping semantic vector; Obtaining text input data based on the non-image modality data; Inputting the mapped semantic vector and the text input data into a pre-trained large language processing model to obtain an answer result for the multimodal question, and outputting the answer result; The visual semantic mapping model is trained using sample data when the parameters of a pre-trained visual extraction model and a pre-trained large language processing model are frozen. The sample data includes multimodal sample question data and answer sample data.

2. The method according to claim 1, characterized in that The imaging modality data includes video data; The method of using a pre-trained visual extraction model to process the image modality data to obtain a visual feature vector includes: Determine key video frames in the video data, and process each of the key video frames using the visual extraction model to obtain extracted visual features of each of the key video frames; The corresponding extracted visual features are sorted according to the order of the key video frames to obtain the visual feature vector.

3. The method according to claim 2, characterized in that The determining of the key video frames in the video data comprises: Using a clustering algorithm to preliminarily extract all video frames in the video data in chronological order to obtain preliminarily screened video frames; A histogram comparison analysis is performed on the preliminary screened video frames to determine key video frames in the preliminary screened video frames.

4. The method according to any one of claims 1 to 3, characterized in that: Before inputting the mapped semantic vector and the text input data into a pre-trained large language processing model, the method further includes: Performing vectorization conversion on the text input data to obtain a search vector; or performing vectorization conversion on the text input data to obtain a conversion vector, and obtaining the search vector based on the conversion vector and a mapping semantic vector; Searching a pre-built domain knowledge vector database based on the search vector to obtain recalled knowledge data; The step of inputting the mapping semantic vector and the text input data into a pre-trained large language processing model to obtain an answer result for the multimodal question includes: inputting the mapping semantic vector, the text input data and the recalled knowledge data into the large language processing model to obtain the answer result.

5. The method according to claim 4, characterized in that The step of inputting the mapping semantic vector, the text input data and the recalled knowledge data into the large language processing model to obtain the answer result for the multimodal question includes: Writing the mapping semantic vector, the text input data and the recalled knowledge data into the content slot corresponding to the pre-constructed prompt template to obtain standard input data; wherein the prompt template includes not only the content slot but also the pre-constructed prompt information; The standard input data is input into the large language processing model to obtain the answer result.

6. The method according to any one of claims 1 to 3, characterized in that: After outputting the answer result, the method further includes: Receive a user's satisfaction rating for the answer result; In a case where the satisfaction score is higher than a satisfaction threshold score, constructing a fine-tuning training sample based on the visual feature vector, the text input data and the answer result; In the case of freezing the parameters of the large language processing model, fine-tuning training is performed on the visual semantic mapping model based on the fine-tuning training samples to obtain a fine-tuned visual semantic mapping model.

7. The method according to claim 6, characterized in that In the case where the satisfaction score is higher than the satisfaction threshold score, the method further comprises: Determining a fine-tuning difference weight corresponding to the fine-tuning training sample based on the satisfaction score, wherein the fine-tuning difference weight is positively correlated with the satisfaction score; The fine-tuning training of the visual semantic mapping model based on the fine-tuning training sample to obtain a fine-tuned visual semantic mapping model includes: Processing the visual feature vector with a visual semantic mapping model to obtain a mapping semantic vector; Inputting the mapped semantic vector and the text input data into a pre-trained large language processing model to obtain an output probability that the model output is the answer result; Based on the output probability and the confirmation probability of the answer result, calculating the loss function corresponding to each fine-tuning training sample; Performing weighted summation based on the fine-tuning difference weights corresponding to each of the fine-tuning training samples and the loss function to obtain a weighted loss function; The weighted loss function is used to fine-tune the visual semantic mapping model to obtain a fine-tuned visual semantic mapping model.

8. A multimodal question answering device in a customer service application scenario, characterized in that: include: An input data acquisition unit, used for acquiring a multimodal question input by a user, wherein the multimodal question includes image modality data and non-image modality data, and the non-image modality data is text data and / or audio data; An image modality data processing unit, configured to process the image modality data using a pre-trained visual extraction model to obtain a visual feature vector; and to process the visual feature vector using a pre-trained visual semantic mapping model to obtain a mapping semantic vector; a non-image modality data removing unit, used for processing the non-image modality data to obtain text input data; A fusion processing unit, configured to input the mapped semantic vector and the text input data into a pre-trained large language processing model, obtain an answer result for the multimodal question and output the answer result; The visual semantic mapping model is trained using sample data when the parameters of a pre-trained visual extraction model and a pre-trained large language processing model are frozen. The sample data includes multimodal sample question data and answer sample data.

9. A computing device, characterized in that It includes a processor and a memory, wherein the memory is used to store a computer program; when the computer program is loaded by the processor, the processor executes the multimodal question answering method in the customer service application scenario as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program. When the computer program is executed by the processor, the processor implements the multimodal question answering method in the customer service application scenario as described in any one of claims 1 to 7.

Images